Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault after consecutive runs #19

Open
brendan-w opened this issue Jun 26, 2015 · 0 comments
Open

Segfault after consecutive runs #19

brendan-w opened this issue Jun 26, 2015 · 0 comments

Comments

@brendan-w
Copy link

Hi,
Thanks for putting together the Etherdream! Always glad to see more FOSS in the laser community. If you get the chance, I could use some help debugging a problem I've run into. I've been trying to work my way through the codebase, but I'm no systems programmer, so I'd appreciate any insight you could provide.

Out of the box, this unit performed very well with the test program in driver/libetherdream. However, after ~20 start/stops of the test program, libetherdream starts reporting negative write counts, and eventually segfaults. I have found that power cycling the DAC resets the problem. I tried updating the firmware to 0.5.0 using your windows updater, but to no avail.

Here's the debug output from the computer, and the DAC's debug header (all of which was generated by running $ ./test). So far, I've been stopping it with a SIGINT.

A successful run (the first run since the DAC was power cycled):

----------
[0.000086]        == libetherdream started ==
[0.000494]        _: listening for DACs...
[0.568173]        _: Found new DAC: 192.168.1.128
[1.200497]        == etherdream_lib_get_dac_count(): 1
0: Ether Dream dc53d5
Connecting...
[1.201455] dc53d5 -- Protocol 0 / LE 0 / playback 1 / source 0
[1.201475] dc53d5 -- Flags: LE 0, playback 0, source 0
[1.201486] dc53d5 -- Buffer: 0 points, 0 pps, 0 total played
[1.201654] dc53d5 DAC version v0.5.0
[1.201775] dc53d5 Ready.
[1.201799] dc53d5 L: waiting
[1.202785] dc53d5 L: st 1 om 0; b 0 + 0 - 0 = 0 -> write 80
[1.202878] dc53d5 L: st 1 om 1; b 0 + 80 - 0 = 80 -> write 80
[1.202923] dc53d5 L: st 1 om 2; b 0 + 160 - 0 = 160 -> write 80
[1.202966] dc53d5 L: st 1 om 3; b 0 + 240 - 0 = 240 -> write 80
[1.203007] dc53d5 L: st 1 om 4; b 0 + 320 - 0 = 320 -> write 80
[1.203048] dc53d5 L: st 1 om 5; b 0 + 400 - 0 = 400 -> write 80
[1.203080] dc53d5 L: st 1 om 6; b 0 + 480 - 0 = 480 -> write 80
[1.203108] dc53d5 L: st 1 om 7; b 0 + 560 - 0 = 560 -> write 40
[1.203134] dc53d5 L: returning to idle
[1.203145] dc53d5 L: waiting
[1.203737] dc53d5 L: st 1 om 8; b 0 + 600 - 0 = 600 -> write 80
[1.203943] dc53d5 L: st 1 om 5; b 240 + 440 - 0 = 680 -> write 80
[1.203975] dc53d5 L: st 1 om 6; b 240 + 520 - 0 = 760 -> write 80
[1.204003] dc53d5 L: st 1 om 7; b 240 + 600 - 0 = 840 -> write 80
[1.204033] dc53d5 L: st 1 om 8; b 240 + 680 - 0 = 920 -> write 80
[1.204271] dc53d5 L: st 1 om 9; b 240 + 760 - 0 = 1000 -> write 80
[1.204485] dc53d5 L: st 1 om 8; b 480 + 600 - 0 = 1080 -> write 80
[1.204623] dc53d5 L: st 1 om 8; b 560 + 600 - 0 = 1160 -> write 40
[1.204689] dc53d5 L: st 1 om 9; b 560 + 640 - 0 = 1200 -> write 80
[1.204796] dc53d5 L: st 1 om 9; b 560 + 720 - 0 = 1280 -> write 80
[1.204837] dc53d5 L: st 1 om 10; b 560 + 800 - 0 = 1360 -> write 80
[1.204874] dc53d5 L: st 1 om 11; b 560 + 880 - 0 = 1440 -> write 80
[1.204962] dc53d5 L: st 1 om 12; b 560 + 960 - 0 = 1520 -> write 80
[1.205028] dc53d5 L: st 1 om 12; b 600 + 1000 - 0 = 1600 -> write 80
[1.206193] dc53d5 L: st 1 om 13; b 600 + 1080 - 0 = 1680 -> write 20
[1.209932] dc53d5 L: Sending begin command...

DAC debug header:

dac 0
dac 0
dhcp: ip 192.168.1.128
dac 0
dac 0
dac 0
dac 0
dac 0
dac 0
dac 0
dac 0
conn
dac: starting
dac 235
dac 235
dac 237
dac 235
dac 235
dac 235
dac 238
dac 235
dac 235
dac 238
dac 238

First error from DAC (happens on run ~15. At this point, the DAC still outputs normally, and client library hasn't failed yet).

dac 238
dac 238
conn
alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 128dac: starting
dac 238
dac 238
dac 238
dac 238
dac 238

A failed run (happens after ~20 runs. The output from both the computer and the DAC gradually grows over consecutive tests.):

----------
[0.000100]        == libetherdream started ==
[0.000520]        _: listening for DACs...
[0.846446]        _: Found new DAC: 192.168.1.128
[1.200517]        == etherdream_lib_get_dac_count(): 1
0: Ether Dream dc53d5
Connecting...
[1.201750] dc53d5 -- Protocol 0 / LE 0 / playback 1 / source 0
[1.201771] dc53d5 -- Flags: LE 0, playback 0, source 0
[1.201784] dc53d5 -- Buffer: 0 points, 0 pps, 0 total played
[1.201952] dc53d5 DAC version v0.5.5
[1.202076] dc53d5 Ready.
[1.202099] dc53d5 L: waiting
[1.203058] dc53d5 L: st 1 om 0; b 0 + 0 - 0 = 0 -> write 80
[1.203148] dc53d5 L: st 1 om 1; b 0 + 80 - 0 = 80 -> write 80
[1.203193] dc53d5 L: st 1 om 2; b 0 + 160 - 0 = 160 -> write 80
[1.203235] dc53d5 L: st 1 om 3; b 0 + 240 - 0 = 240 -> write 80
[1.203276] dc53d5 L: st 1 om 4; b 0 + 320 - 0 = 320 -> write 80
[1.203317] dc53d5 L: st 1 om 5; b 0 + 400 - 0 = 400 -> write 80
[1.203347] dc53d5 L: st 1 om 6; b 0 + 480 - 0 = 480 -> write 80
[1.203376] dc53d5 L: st 1 om 7; b 0 + 560 - 0 = 560 -> write 40
[1.203403] dc53d5 L: returning to idle
[1.203414] dc53d5 L: waiting
[1.204062] dc53d5 L: st 1 om 8; b 0 + 600 - 0 = 600 -> write 80
[1.204212] dc53d5 L: st 1 om 6; b 160 + 520 - 0 = 680 -> write 80
[1.204266] dc53d5 L: st 1 om 7; b 160 + 600 - 0 = 760 -> write 80
[1.204333] dc53d5 L: st 1 om 8; b 160 + 680 - 0 = 840 -> write 80
[1.204392] dc53d5 L: st 1 om 9; b 160 + 760 - 0 = 920 -> write 80
[1.204977] dc53d5 L: st 1 om 10; b 160 + 840 - 0 = 1000 -> write 80
[1.205062] dc53d5 L: st 1 om 10; b 240 + 840 - 0 = 1080 -> write 80
[1.205102] dc53d5 L: st 1 om 11; b 240 + 920 - 0 = 1160 -> write 40
[1.205151] dc53d5 L: st 1 om 12; b 240 + 960 - 0 = 1200 -> write 80
[1.206014] dc53d5 L: st 1 om 13; b 240 + 1040 - 0 = 1280 -> write 80
[1.206062] dc53d5 L: st 1 om 14; b 240 + 1120 - 0 = 1360 -> write 80
[1.206102] dc53d5 L: st 1 om 15; b 240 + 1200 - 0 = 1440 -> write 80
[1.206141] dc53d5 L: st 1 om 16; b 240 + 1280 - 0 = 1520 -> write 80
[1.206194] dc53d5 L: st 1 om 17; b 240 + 1360 - 0 = 1600 -> write 80
[1.215163] dc53d5 L: st 1 om 18; b 240 + 1440 - 0 = 1680 -> write 20
[1.216385] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.217490] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.218590] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.219684] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.220781] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.221876] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.222970] dc53d5 L: st 1 om 18; b 320 + 1380 - 0 = 1700 -> write 0
[1.224096] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.225191] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.226286] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.227383] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.228478] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.229570] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.230664] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.231781] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.232939] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.234053] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.235207] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.236342] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.237436] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.238526] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.239614] dc53d5 L: st 1 om 17; b 480 + 1300 - 0 = 1780 -> write -80
[1.240798] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.241889] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.242977] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.244064] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.245152] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.246239] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.247327] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.248416] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.249502] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.250612] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.251708] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.252803] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.253901] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.254995] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.256089] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.257183] dc53d5 L: st 1 om 15; b 641 + 1140 - 0 = 1781 -> write -81
[1.350436] dc53d5 L: Sending begin command...
fish: Job 1, “./test” terminated by signal SIGSEGV (Address boundary error)

DAC debug header:

dac 238
dac 235
dac 235
conn
alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
1 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
2 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
4 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
4 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 2 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
4 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
4 deferred acks
alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 2 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
4 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 70: oom in 128alloc 70: oom in 3844 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
6 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
6 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 2 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
8 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
8 deferred acks
alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 2 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
8 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
8 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
10 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
10 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 2 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
10 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
deferring ACK
11 deferred acks
alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 0 deferred acks, err 0
alloc 70: oom in 128alloc 70: oom in 384dac: starting
11 deferred acks
alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 128alloc 92: oom in 384tcp_enq: alloc pbuf
sent 8 deferred acks, err 0
close_conn: 1
dac 238
dac 238

gdb trace from the fault, which, not suprisingly, takes place at a memcpy.

(gdb) where
#0  0x00007ffff739e550 in __memcpy_sse2_unaligned () from /usr/lib/libc.so.6
#1  0x0000000000402261 in dac_send_data (d=0x7ffff00008c0, data=0x7fffeffcdf16, npoints=58, rate=30000) at etherdream.c:490
#2  0x0000000000402677 in dac_loop (dv=0x7ffff00008c0) at etherdream.c:605
#3  0x00007ffff78c0374 in start_thread () from /usr/lib/libpthread.so.0
#4  0x00007ffff73f627d in clone () from /usr/lib/libc.so.6
@brendan-w brendan-w changed the title Segfault after multiple consequtive runs Segfault after multiple consecutive runs Jun 26, 2015
@brendan-w brendan-w changed the title Segfault after multiple consecutive runs Segfault after consecutive runs Jun 26, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@brendan-w