hthread is a thread error detector and helper library with synchronization errors detection support for c/c++ programs that use the pthreads.
hthread is a thread error detector and helper library with synchronization errors detection support for c/c++ programs that use the pthreads.
hthread is a lightweight tool for detecting synchronization errors in c/c++ applications, specifically designed for embedded systems.
main use case may include embedded systems where valgrind - drd, or helgrind support is not available.
and has benefits of:
- has a negligible effect run-time speed
- does not require any source code change
- operating system and architecture independent
- easy to use
can detect errors of:
hthread is able to detect and report following errors. although some of them are too obvious, early detection is much better than to deal with hard-to-find bugs.
- destroying an invalid mutex - fail-00.c, report
- locking an invalid mutex - fail-01.c, report
- unlocking an invalid mutex - fail-02.c, report
- locking an already locked mutex - fail-03.c, report
- unlocking an unheld mutex - fail-05.c, report
- destroying a locked mutex - fail-06.c, report
- destroying an invalid condition - fail-20.c, report
- signaling an invalid condition - fail-21.c, report
- broadcasting an invalid condition - fail-22.c, report
- [timed]waiting on an invalid condition - fail-23.c, report
- [timed]waiting on an invalid mutex - fail-24.c, report
- [timed]waiting on an unheld mutex - fail-25.c, report
- join invalid thread - fail-40.c, report
- detach invalid thread - fail-41.c, report
- unlocking mutex that was held by other thread - fail-42.c, report
hthread is able to detect inconsistent locking orders, it is very useful because they usually result to deadlocks. they may never be discovered during testing and lead to hard-to-find bugs.
hthread monitors locks/unlocks and stores them in separate tables for each thread. this allows hthread to build a global lock order table for running process. hthread checks for locking order violation against global order table for each lock request, and able to detect and report following errors.
- lock order violation in same thread - fail-04.c, report
- lock order violation while [timed]waiting on condition - fail-26.c, report
- lock order violation between threads - fail-43.c, report
hthread able to understand and print report about lock contentions - a thread has to wait until requested lock is released. monitoring lock contentions is handy because, they usually cause unwanted delays or they may point to an undetected potential deadlock. hthread can report followings.
- waiting to lock a mutex more than allowed threshold - fail-60.c, report
- hold a mutex lock more than allowed threshold - fail-61.c, report
hthread configuration parameters can be set using make flags, please check example section for demonstration.
-
HTHREAD_ENABLE_CALLSTACK
default 1
enable/disable reporting call trace information on error, useful but depends on libbdf, libdl, and backtrace function from glibc. may be disabled for toolchains which does not support backtracing.
-
HTHREAD_REPORT_CALLSTACK
default 1
dump callstack info (function call history) for error point.
-
HTHREAD_ASSERT_ON_ERROR
default 1
terminate the process on any pthreads api misuse and/or lock order violation.
-
HTHREAD_LOCK_THRESHOLD
default 5000 miliseconds
print report if lock is held longer than the specified time, in miliseconds.
-
HTHREAD_LOCK_THRESHOLD_ASSERT
default 0
terminate if lock is held longer than the specified time, in miliseconds.
-
HTHREAD_LOCK_TRY_THRESHOLD
default 5000 miliseconds
print report if locking operation takes longer than the specified time, in miliseconds.
-
HTHREAD_LOCK_TRY_THRESHOLD_ASSERT
default 0
terminate if locking operation takes longer than the specified time, in miliseconds.
hthread reads configuration parameters from environment via getenv function call. one can either set/change environment variables in source code of monitored project via setenv function call, or set them globally in running shell using export function.
please check example section for demonstration.
-
hthread_report_callstack
default 1
dump callstack info (function call history) for error point.
-
hthread_assert_on_error
default 1
terminate the process on any pthreads api misuse and/or lock order violation.
-
hthread_lock_hreashold
default 5000 miliseconds
print report if lock is held longer than the specified time, in miliseconds.
-
hthread_lock_hreashold_assert
default 0
terminate if lock is held longer than the specified time, in miliseconds.
-
hthread_lock_try_threshold
default 5000 miliseconds
print report if locking operation takes longer than the specified time, in miliseconds.
-
hthread_lock_try_threshold_assert
default 0
terminate if locking operation takes longer than the specified time, in miliseconds.
- destroying an invalid mutex
- locking an invalid mutex
- unlocking an invalid mutex
- locking an already locked mutex
- unlocking an unheld mutex
- destroying a locked mutex
- destroying an invalid condition
- signaling an invalid condition
- broadcasting an invalid condition
- [timed]waiting on an invalid condition
- [timed]waiting on an invalid mutex
- [timed]waiting on an unheld mutex
- join invalid thread
- detach invalid thread
- unlocking mutex that was held by other thread
example code: fail-00.c
# ./test/fail-00-debug
(hthread:30786) new thread created: 'root-process (0x1427010)'
(hthread:30786) at: (null) (null):0
(hthread:30786) mutex destroy with invalid mutex: '0x7fff1476d258'
(hthread:30786) by: root-process (0x1427010)
(hthread:30786) at: main fail-00.c:23
(hthread:30786) 0x40c12f: hthread.c (debug_dump_callstack:829)
(hthread:30786) 0x401b5f: fail-00.c (main:24)
(hthread:30786) 0x401bbd: (null) (_start:0)
fail-00-debug: hthread.c:1349: debug_mutex_del: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.
example code: fail-01.c
# ./test/fail-01-debug
(hthread:30789) new thread created: 'root-process (0x1499010)'
(hthread:30789) at: (null) (null):0
(hthread:30789) mutex lock with invalid mutex: '0x7fff394852f8'
(hthread:30789) by: root-process (0x1499010)
(hthread:30789) at: main fail-01.c:23
(hthread:30789) 0x408e83: hthread.c (debug_dump_callstack:829)
(hthread:30789) 0x40c2f4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30789) 0x401b5f: fail-01.c (main:24)
(hthread:30789) 0x401bbd: (null) (_start:0)
fail-01-debug: hthread.c:946: debug_mutex_add_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.
example code: fail-02.c
# ./test/fail-02-debug
(hthread:30792) new thread created: 'root-process (0x1f9e010)'
(hthread:30792) at: (null) (null):0
(hthread:30792) mutex unlock with invalid mutex '0x7ffffa879c38'
(hthread:30792) by: root-process (0x1f9e010)
(hthread:30792) at: main fail-02.c:23
(hthread:30792) 0x406a7c: hthread.c (debug_dump_callstack:829)
(hthread:30792) 0x40d619: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30792) 0x401b5f: fail-02.c (main:24)
(hthread:30792) 0x401bbd: (null) (_start:0)
fail-02-debug: hthread.c:1226: debug_mutex_del_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.
example code: fail-03.c
# ./test/fail-03-debug
(hthread:30795) new thread created: 'root-process (0x1e670c0)'
(hthread:30795) at: (null) (null):0
(hthread:30795) mutex lock with already held mutex: 'mutex(main fail-03.c:22) (0x1e67010)'
(hthread:30795) by: root-process (0x1e670c0)
(hthread:30795) at: main fail-03.c:32
(hthread:30795) 0x408ef2: hthread.c (debug_dump_callstack:829)
(hthread:30795) 0x40c4d4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30795) 0x401cb8: fail-03.c (main:33)
(hthread:30795) 0x401da1: (null) (_start:0)
(hthread:30795) previously acquired
(hthread:30795) by: root-process (0x1e670c0)
(hthread:30795) at: main fail-03.c:27
(hthread:30795) created 'mutex(main fail-03.c:22) (0x1e67010)'
(hthread:30795) at: main fail-03.c:22
fail-03-debug: hthread.c:963: debug_mutex_add_lock: Assertion `(mt == ((void *)0)) && "mutex is already locked"' failed.
example code: fail-05.c
# ./test/fail-05-debug
(hthread:30801) new thread created: 'root-process (0x9200c0)'
(hthread:30801) at: (null) (null):0
(hthread:30801) mutex unlock with unheld mutex: 'mutex(main fail-05.c:22) (0x920010)'
(hthread:30801) by: root-process (0x9200c0)
(hthread:30801) at: main fail-05.c:27
(hthread:30801) 0x406f94: hthread.c (debug_dump_callstack:829)
(hthread:30801) 0x40d7d9: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30801) 0x401c95: fail-05.c (main:28)
(hthread:30801) 0x401d81: (null) (_start:0)
(hthread:30801) created 'mutex(main fail-05.c:22) (0x920010)'
(hthread:30801) at: main fail-05.c:22
fail-05-debug: hthread.c:1259: debug_mutex_del_lock: Assertion `(mtl != ((void *)0)) && "mutex is not locked"' failed.
example code: fail-06.c
# ./test/fail-06-debug
(hthread:30804) new thread created: 'root-process (0x6420c0)'
(hthread:30804) at: (null) (null):0
(hthread:30804) mutex destroy with currently locked mutex: '0x642010'
(hthread:30804) by: root-process (0x6420c0)
(hthread:30804) at: main fail-06.c:32
(hthread:30804) 0x40c21c: hthread.c (debug_dump_callstack:829)
(hthread:30804) 0x401cb8: fail-06.c (main:33)
(hthread:30804) 0x401d81: (null) (_start:0)
(hthread:30804) lock observed
(hthread:30804) by: root-process (0x6420c0)
(hthread:30804) at: main fail-06.c:27
(hthread:30804) created 'mutex(main fail-06.c:22) (0x642010)
(hthread:30804) at: main fail-06.c:22
fail-06-debug: hthread.c:1366: debug_mutex_del: Assertion `(mt == ((void *)0)) && "invalid mutex"' failed.
example code: fail-20.c
# ./test/fail-20-debug
(hthread:30807) new thread created: 'root-process (0x1829010)'
(hthread:30807) at: (null) (null):0
(hthread:30807) cond destroy with invalid condition: '0x7fff491eb5a8'
(hthread:30807) by: root-process (0x1829010)
(hthread:30807) at: main fail-20.c:23
(hthread:30807) 0x40a413: hthread.c (debug_dump_callstack:829)
(hthread:30807) 0x401b5f: fail-20.c (main:24)
(hthread:30807) 0x401bbd: (null) (_start:0)
fail-20-debug: hthread.c:1436: debug_cond_del: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.
example code: fail-21.c
# ./test/fail-21-debug
(hthread:30810) new thread created: 'root-process (0x2019010)'
(hthread:30810) at: (null) (null):0
(hthread:30810) cond signal with invalid condition: '0x7fffd0b4ec78'
(hthread:30810) by: root-process (0x2019010)
(hthread:30810) at: main fail-21.c:23
(hthread:30810) 0x405769: hthread.c (debug_dump_callstack:829)
(hthread:30810) 0x40a5a9: hthread.c (hthread_cond_signal_actual_debug:457)
(hthread:30810) 0x401b5f: fail-21.c (main:24)
(hthread:30810) 0x401c01: (null) (_start:0)
fail-21-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.
example code: fail-22.c
# ./test/fail-22-debug
(hthread:30813) new thread created: 'root-process (0x10b8010)'
(hthread:30813) at: (null) (null):0
(hthread:30813) cond signal with invalid condition: '0x7fffafb97db8'
(hthread:30813) by: root-process (0x10b8010)
(hthread:30813) at: main fail-22.c:23
(hthread:30813) 0x405769: hthread.c (debug_dump_callstack:829)
(hthread:30813) 0x40a5d9: hthread.c (hthread_cond_broadcast_actual_debug:463)
(hthread:30813) 0x401b5f: fail-22.c (main:24)
(hthread:30813) 0x401c01: (null) (_start:0)
fail-22-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.
example code: fail-23.c
# ./test/fail-23-debug
(hthread:30816) new thread created: 'root-process (0x10340c0)'
(hthread:30816) at: (null) (null):0
(hthread:30816) cond timedwait with invalid condition: '0x7fffaab0c968'
(hthread:30816) by: root-process (0x10340c0)
(hthread:30816) at: main fail-23.c:43
(hthread:30816) 0x405a09: hthread.c (debug_dump_callstack:829)
(hthread:30816) 0x40a8db: hthread.c (hthread_cond_timedwait_tspec_actual_debug:495)
(hthread:30816) 0x401cf4: fail-23.c (main:43)
(hthread:30816) 0x401ea5: (null) (_start:0)
fail-23-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.
example code: fail-24.c
# ./test/fail-24-debug
(hthread:30819) new thread created: 'root-process (0x1e320c0)'
(hthread:30819) at: (null) (null):0
(hthread:30819) cond timedwait with invalid mutex '0x7fff3e5d79d8'
(hthread:30819) by: root-process (0x1e320c0)
(hthread:30819) at: main fail-24.c:38
(hthread:30819) 0x406d1c: hthread.c (debug_dump_callstack:829)
(hthread:30819) 0x40a8b3: hthread.c (hthread_cond_timedwait_tspec_actual_debug:496)
(hthread:30819) 0x401cd1: fail-24.c (main:38)
(hthread:30819) 0x401e59: (null) (_start:0)
fail-24-debug: hthread.c:1226: debug_mutex_del_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.
example code: fail-25.c
# ./test/fail-25-debug
(hthread:30822) new thread created: 'root-process (0x23af0c0)'
(hthread:30822) at: (null) (null):0
(hthread:30822) cond timedwait with unheld mutex: 'mutex(main fail-25.c:32) (0x23afa00)'
(hthread:30822) by: root-process (0x23af0c0)
(hthread:30822) at: main fail-25.c:52
(hthread:30822) 0x407184: hthread.c (debug_dump_callstack:829)
(hthread:30822) 0x40a9c3: hthread.c (hthread_cond_timedwait_tspec_actual_debug:496)
(hthread:30822) 0x401d72: fail-25.c (main:52)
(hthread:30822) 0x401f6d: (null) (_start:0)
(hthread:30822) created 'mutex(main fail-25.c:32) (0x23afa00)'
(hthread:30822) at: main fail-25.c:32
fail-25-debug: hthread.c:1259: debug_mutex_del_lock: Assertion `(mtl != ((void *)0)) && "mutex is not locked"' failed.
example code: fail-40.c
# ./test/fail-40-debug
(hthread:30828) new thread created: 'root-process (0xe20010)'
(hthread:30828) at: (null) (null):0
(hthread:30828) join with invalid thread: '0x7fffecd84218'
(hthread:30828) by: root-process (0xe20010)
(hthread:30828) at: main fail-40.c:25
(hthread:30828) 0x404ff5: hthread.c (debug_dump_callstack:829)
(hthread:30828) 0x40daee: hthread.c (hthread_join_actual_debug:690)
(hthread:30828) 0x401b5f: fail-40.c (main:26)
(hthread:30828) 0x401bbd: (null) (_start:0)
fail-40-debug: hthread.c:349: hthread_check: Assertion `(th == thread) && "invalid thread"' failed.
example code: fail-41.c
# ./test/fail-41-debug
(hthread:30831) new thread created: 'root-process (0x258a010)'
(hthread:30831) at: (null) (null):0
(hthread:30831) detach with invalid thread: '0x7fff81b0b8d8'
(hthread:30831) by: root-process (0x258a010)
(hthread:30831) at: main fail-41.c:25
(hthread:30831) 0x404ff5: hthread.c (debug_dump_callstack:829)
(hthread:30831) 0x40db7e: hthread.c (hthread_detach_actual_debug:704)
(hthread:30831) 0x401b5f: fail-41.c (main:26)
(hthread:30831) 0x401bbd: (null) (_start:0)
fail-41-debug: hthread.c:349: hthread_check: Assertion `(th == thread) && "invalid thread"' failed.
example code: fail-42.c
# ./test/fail-42-debug
(hthread:30834) new thread created: 'root-process (0x1e560c0)'
(hthread:30834) at: (null) (null):0
(hthread:30834) new thread created: 'thread(main fail-42.c:48) (0x1e56cd0)'
(hthread:30834) at: main fail-42.c:48
(hthread:30834) mutex unlock with a mutex 'mutex(main fail-42.c:38) (0x1e56010)' currently hold by other thread
(hthread:30834) by: thread(main fail-42.c:48) (0x1e56cd0)
(hthread:30834) at: worker fail-42.c:23
(hthread:30834) 0x406d9d: hthread.c (debug_dump_callstack:829)
(hthread:30834) 0x40d929: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30834) 0x401f4f: fail-42.c (worker:24)
(hthread:30834) 0x40da0a: hthread.c (thread_run:619)
(hthread:30834) 0x7f51f6de1e9a: pthread_create.c (start_thread:308)
(hthread:30834) lock observed
(hthread:30834) by: root-process (0x1e560c0)
(hthread:30834) at: main fail-42.c:43
(hthread:30834) created 'mutex(main fail-42.c:38) (0x1e56010)'
(hthread:30834) at: main fail-42.c:38
fail-42-debug: hthread.c:1247: debug_mutex_del_lock: Assertion `(mtl == ((void *)0)) && "mutex is locked by other thread"' failed.
- lock order violation in same thread
- lock order violation while [timed]waiting on condition
- lock order violation between threads
example code: fail-04.c
# ./test/fail-04-debug
(hthread:30798) new thread created: 'root-process (0x1d530c0)'
(hthread:30798) at: (null) (null):0
(hthread:30798) mutex lock order 'mutex(main fail-04.c:26) (0x1d53b60)' before 'mutex(main fail-04.c:26) (0x1d53c10)' violated
(hthread:30798) incorrect order is: acquisition of 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798) by: root-process (0x1d530c0)
(hthread:30798) at: main fail-04.c:47
(hthread:30798) followed by a later acquisition of 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798) by: root-process (0x1d530c0)
(hthread:30798) at: main fail-04.c:47
(hthread:30798) 0x408b94: hthread.c (debug_dump_callstack:829)
(hthread:30798) 0x40c5c4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30798) 0x401d20: fail-04.c (main:48)
(hthread:30798) 0x401e89: (null) (_start:0)
(hthread:30798) required order is: acquisition of 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798) by: root-process (0x1d530c0)
(hthread:30798) at: main fail-04.c:33
(hthread:30798) followed by a later acquisition of 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798) by: root-process (0x1d530c0)
(hthread:30798) at: main fail-04.c:33
(hthread:30798) created 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798) at: main fail-04.c:26
(hthread:30798) created 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798) at: main fail-04.c:26
fail-04-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.
example code: fail-26.c
# ./test/fail-26-debug
(hthread:30825) new thread created: 'root-process (0x77b0c0)'
(hthread:30825) at: (null) (null):0
(hthread:30825) cond wait order 'mutex(main fail-26.c:32) (0x77b010)' before 'mutex(main fail-26.c:32) (0x77ba00)' violated
(hthread:30825) incorrect order is: acquisition of 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825) by: root-process (0x77b0c0)
(hthread:30825) at: main fail-26.c:48
(hthread:30825) followed by a later acquisition of 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825) by: root-process (0x77b0c0)
(hthread:30825) at: main fail-26.c:57
(hthread:30825) 0x408cb4: hthread.c (debug_dump_callstack:829)
(hthread:30825) 0x40aa3b: hthread.c (hthread_cond_timedwait_tspec_actual_debug:513)
(hthread:30825) 0x401d95: fail-26.c (main:57)
(hthread:30825) 0x401fb5: (null) (_start:0)
(hthread:30825) required order is: acquisition of 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825) by: root-process (0x77b0c0)
(hthread:30825) at: main fail-26.c:43
(hthread:30825) followed by a later acquisition of 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825) by: root-process (0x77b0c0)
(hthread:30825) at: main fail-26.c:48
(hthread:30825) created 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825) at: main fail-26.c:32
(hthread:30825) created 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825) at: main fail-26.c:32
fail-26-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.
example code: fail-43.c
# ./test/fail-43-debug
(hthread:30838) new thread created: 'root-process (0x14ae0c0)'
(hthread:30838) at: (null) (null):0
(hthread:30838) new thread created: 'thread(main fail-43.c:73) (0x14af0f0)'
(hthread:30838) at: main fail-43.c:73
(hthread:30838) mutex lock order 'mutex(main fail-43.c:53) (0x14ae010)' before 'mutex(main fail-43.c:58) (0x14aea00)' violated
(hthread:30838) incorrect order is: acquisition of 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838) by: thread(main fail-43.c:73) (0x14af0f0)
(hthread:30838) at: worker fail-43.c:23
(hthread:30838) followed by a later acquisition of 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838) by: thread(main fail-43.c:73) (0x14af0f0)
(hthread:30838) at: worker fail-43.c:28
(hthread:30838) 0x408d24: hthread.c (debug_dump_callstack:829)
(hthread:30838) 0x40c754: hthread.c (debug_mutex_try_lock:1088)
(hthread:30838) 0x40206f: fail-43.c (worker:29)
(hthread:30838) 0x40db5a: hthread.c (thread_run:619)
(hthread:30838) 0x7f58cbd27e9a: pthread_create.c (start_thread:308)
(hthread:30838) required order is: acquisition of 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838) by: root-process (0x14ae0c0)
(hthread:30838) at: main fail-43.c:63
(hthread:30838) followed by a later acquisition of 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838) by: root-process (0x14ae0c0)
(hthread:30838) at: main fail-43.c:68
(hthread:30838) created 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838) at: main fail-43.c:58
(hthread:30838) created 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838) at: main fail-43.c:53
fail-43-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.
example code: fail-60.c
lock try threshold is set to 1 seconds, and enabled terminating if waited more than threshold time.
# hthread_lock_try_threshold=1000 hthread_lock_try_threshold_assert=1 ./test/fail-60-debug
(hthread:30880) new thread created: 'root-process (0x13670c0)'
(hthread:30880) at: (null) (null):0
(hthread:30880) new thread created: 'thread(main fail-60.c:53) (0x1367cd0)'
(hthread:30880) at: main fail-60.c:53
(hthread:30880) mutex lock still waiting for mutex: 'mutex(main fail-60.c:43) (0x1367010)'
(hthread:30880) by: thread(main fail-60.c:53) (0x1367cd0)
(hthread:30880) at: worker fail-60.c:23
(hthread:30880) 0x40d6e5: hthread.c (debug_dump_callstack:829)
(hthread:30880) 0x401fbf: fail-60.c (worker:24)
(hthread:30880) 0x40da8a: hthread.c (thread_run:619)
(hthread:30880) 0x7f5bf1a76e9a: pthread_create.c (start_thread:308)
(hthread:30880) currently locked
(hthread:30880) by: root-process (0x13670c0)
(hthread:30880) at: main fail-60.c:48
(hthread:30880) currently locked
(hthread:30880) by: thread(main fail-60.c:53) (0x1367cd0)
(hthread:30880) at: worker fail-60.c:23
(hthread:30880) created 'mutex(main fail-60.c:43) (0x1367010)'
(hthread:30880) at: main fail-60.c:43
fail-60-debug: hthread.c:1166: debug_mutex_try_lock: Assertion `(a == 0) && "mutex try lock threshold reached"' failed.
example code: fail-61.c
lock hold threshold is set to 1 seconds, and enabled terminating if waited more than threshold time.
# hthread_lock_threshold=1000 hthread_lock_threshold_assert=1 ./test/fail-61-debug
(hthread:30885) new thread created: 'root-process (0x15a20c0)'
(hthread:30885) at: (null) (null):0
(hthread:30885) mutex unlock with a mutex 'mutex(main fail-61.c:22) (0x15a2010)' hold during 3001 ms
(hthread:30885) by: root-process (0x15a20c0)
(hthread:30885) at: main fail-61.c:33
(hthread:30885) 0x406bc6: hthread.c (debug_dump_callstack:829)
(hthread:30885) 0x40d849: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30885) 0x401d06: fail-61.c (main:34)
(hthread:30885) 0x401ded: (null) (_start:0)
(hthread:30885) lock observed
(hthread:30885) by: root-process (0x15a20c0)
(hthread:30885) at: main fail-61.c:27
(hthread:30885) created 'mutex(main fail-61.c:22) (0x15a2010)'
(hthread:30885) at: main fail-61.c:22
fail-61-debug: hthread.c:1286: debug_mutex_del_lock: Assertion `(a == 0) && "mutex lock threshold reached"' failed.
| test | sucess | fail |
|---|---|---|
|
sucess-00.c fail-00.c |
init mutex destroy mutex |
destroy mutex ** invalid mutex ** |
|
sucess-01.c fail-01.c |
init mutex lock mutex unlock mutex destroy mutex |
lock mutex ** invalid mutex ** |
|
sucess-02.c fail-02.c |
init mutex lock mutex unlock mutex destroy mutex |
unlock mutex ** invalid mutex ** |
|
sucess-03.c fail-03.c |
init mutex lock mutex unlock mutex destroy mutex |
init mutex lock mutex lock mutex ** mutex is already locked ** |
|
sucess-04.c fail-04.c |
init mutexes lock mutexes 0...n unlock mutexes 0...n lock mutexes 0...n unlock mutexes 0...n destroy mutexes |
init mutexes lock mutexes 0...n unlock mutexes 0...n lock mutexes n...0 ** lock order violation ** |
|
sucess-05.c fail-05.c |
init mutex lock mutex unlock mutex destroy mutex |
init mutex unlock mutex ** unlocking a unheld mutex ** |
|
sucess-06.c fail-06.c |
init mutex lock mutex unlock mutex destroy mutex |
init mutex lock mutex destroy mutex ** destroying a locked mutex ** |
| test | sucess | fail |
|---|---|---|
|
sucess-20.c fail-20.c |
init condition destroy condition |
destroy condition ** invalid condition ** |
|
sucess-21.c fail-21.c |
init condition signal condition destroy condition |
signal condition ** invalid condition ** |
|
sucess-22.c fail-22.c |
init condition broadcast condition destroy condition |
broadcast condition ** invalid condition ** |
|
sucess-23.c fail-23.c |
init condition init mutex lock mutex timed wait on condition, mutex unlock mutex destroy condition destroy mutex |
init mutex lock mutex timed wait on condition, mutex ** invalid condition ** |
|
sucess-24.c fail-24.c |
init condition init mutex lock mutex timed wait on condition, mutex unlock mutex destroy condition destroy mutex |
init condition timed wait on condition, mutex ** invalid mutex ** |
|
sucess-25.c fail-25.c |
init mutexes init condition lock mutex 0 timed wait on condition, mutex 0 unlock mutex 0 destroy condition destroy mutexes |
init mutexes init condition lock mutex 0 timed wait on condition, mutex 1 ** mutex not locked ** |
|
sucess-26.c fail-26.c |
init mutexes init condition lock mutex 0 lock mutex 1 unlock mutex 1 timed wait on condition, mutex 0 unlock mutex 0 destroy condition destroy mutexes |
init mutexes init condition lock mutex 0 lock mutex 1 timed wait on condition, mutex 0 ** lock order will be violated ** |
| test | sucess | fail |
|---|---|---|
|
sucess-40.c fail-40.c |
create thread join thread |
join thread ** invalid thread ** |
|
sucess-41.c fail-41.c |
create thread detach thread |
detach thread ** invalid thread ** |
|
sucess-42.c fail-42.c |
main : init mutex main : lock mutex main : create thread thread: lock mutex thread: unlock mutex main : unlock mutex main : join thread main : destroy mutex |
main : init mutex main : lock mutex main : create thread thread: unlock mutex ** unlocking mutex currently held ** ** by other thread ** |
|
sucess-43.c fail-43.c |
main : init mutex 0 main : init mutex 1 main : lock mutex 0 main : lock mutex 1 main : create thread thread: lock mutex 0 thread: lock mutex 1 thread: unlock mutex 0 thread: unlock mutex 1 main : unlock mutex 0 main : unlock mutex 1 main : join thread main : destroy mutex |
main : init mutex 0 main : init mutex 1 main : lock mutex 0 main : lock mutex 1 main : create thread thread: lock mutex 1 thread: lock mutex 0 ** lock order violation ** |
| test | sucess | fail |
|---|---|---|
|
sucess-60.c fail-60.c |
hthread_lock_try_threshold = 1000 hthread_lock_try_threshold_assert = 0 main : init mutex main : lock mutex main : create thread main : sleep 3 thread: lock mutex ** try lock threshold reached ** ** still waiting on mutex ** thread: unlock mutex main : unlock mutex main : join thread main : destroy mutex |
hthread_lock_try_threshold = 1000 hthread_lock_try_threshold_assert = 1 main : init mutex main : lock mutex main : create thread main : sleep 3 thread: lock mutex ** try lock threshold reached ** ** still waiting on mutex ** ** assert requested ** |
|
sucess-61.c fail-61.c |
hthread_lock_threshold = 1000 hthread_lock_threshold_assert = 0 init mutex lock mutex sleep 3 unlock mutex ** lock threshold reached ** destroy mutex |
hthread_lock_threshold = 1000 hthread_lock_assert = 1 init mutex lock mutex sleep 3 unlock mutex ** lock threshold reached ** ** assert requested ** |
using hthread is pretty simple, just clone libhthread and build;
- add -include hthread.h -DHTHREAD_DEBUG=1 -g -O1 to target cflags
- link with -lhthread -lrt if HTHREAD_ENABLE_CALLSTACK is 0 or
- link with -lhthread -lrt -ldl -lbfd if HTHREAD_ENABLE_CALLSTACK is 1
compile libhthread with callstack support
# git clone git://github.com/anhanguera/libhthread.git
# cd libhthread
# HTHREAD_ENABLE_CALLSTACK=1 make
compile libhthread without callstack support
# git clone git://github.com/anhanguera/libhthread.git
# cd libhthread
# HTHREAD_ENABLE_CALLSTACK=0 make
let below is the source code - with double lock error - to be monitored:
1 #include <stdio.h>
2 #include <stdlib.h>
3 #include <unistd.h>
4 #include <pthread.h>
5
6 int main (int argc, char *argv[])
7 {
8 int rc;
9 pthread_mutex_t m;
10 (void) argc;
11 (void) argv;
12 rc = pthread_mutex_init(&m, NULL);
13 if (rc != 0) {
14 fprintf(stderr, "pthread_mutex_init failed\n");
15 exit(-1);
16 }
17 rc = pthread_mutex_lock(&m);
18 if (rc != 0) {
19 fprintf(stderr, "pthread_mutex_lock failed\n");
20 exit(-1);
21 }
22 rc = pthread_mutex_lock(&m);
23 if (rc != 0) {
24 fprintf(stderr, "pthread_mutex_lock failed\n");
25 exit(-1);
26 }
27 rc = pthread_mutex_destroy(&m);
28 if (rc != 0) {
29 fprintf(stderr, "pthread_mutex_destroy failed\n");
30 exit(-1);
31 }
32 return 0;
32 }
compile and run as usual:
# gcc -o app main.c -lpthread
# ./app
application will not exit, because it is trying to lock a already locked mutex. now, enable monitoring with hthread:
# gcc -include src/hthread.h -DHTHREAD_DEBUG=1 -g -O1 -o app-debug main.c -lhthread -lrt -ldl -lbfd -lpthread
# ./app-debug
(hthread:30967) new thread created: 'root-process (0x11850b0)'
(hthread:30967) at: (null) (null):0
(hthread:30967) mutex lock with already held mutex: 'mutex(main main.c:12) (0x1185010)'
(hthread:30967) by: root-process (0x11850b0)
(hthread:30967) at: main main.c:22
(hthread:30967) 0x7f744a1d0d52: hthread.c (debug_dump_callstack:829)
(hthread:30967) 0x7f744a1d4334: hthread.c (debug_mutex_try_lock:1088)
(hthread:30967) 0x400986: main.c (main:23)
(hthread:30967) 0x4007e9: (null) (_start:0)
(hthread:30967) previously acquired
(hthread:30967) by: root-process (0x11850b0)
(hthread:30967) at: main main.c:17
(hthread:30967) created 'mutex(main main.c:12) (0x1185010)'
(hthread:30967) at: main main.c:12
app-debug: hthread.c:963: debug_mutex_add_lock: Assertion `(mt == ((void *)0)) && "mutex is already locked"' failed.
hthread detected and reported the error: application was trying to lock an already locked mutex at line 22, which was previously locked at line 17, and was created at line 12. and terminated the process.
program termination on error can be disabled with hthread_assert_on_error configuration parameter
# hthread_assert_on_error=0 ./app-debug
(hthread:32648) new thread created: 'root-process (0xab10b0)'
(hthread:32648) at: (null) (null):0
(hthread:32648) mutex lock with already held mutex: 'mutex(main main.c:12) (0xab1010)'
(hthread:32648) by: root-process (0xab10b0)
(hthread:32648) at: main main.c:22
(hthread:32648) previously acquired
(hthread:32648) by: root-process (0xab10b0)
(hthread:32648) at: main main.c:17
(hthread:32648) created 'mutex(main main.c:12) (0xab1010)'
(hthread:32648) at: main main.c:12
hthread::error: (mt == NULL) && "mutex is already locked" (debug_mutex_add_lock hthread.c:823)
this time hthread detected and reported the error, but not terminated the process.
if you are using the software and/or have any questions, suggestions, etc. please contact with me at [email protected]
Copyright (C) 2009-2013 Alper Akcan [email protected]
This work is free. It comes without any warranty, to the extent permitted by applicable law. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See the COPYING file for more details.