Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault on Ubuntu 24.04 accessing blob #46

Open
aaronwmorris opened this issue Jun 25, 2024 · 5 comments
Open

Segfault on Ubuntu 24.04 accessing blob #46

aaronwmorris opened this issue Jun 25, 2024 · 5 comments
Assignees
@pawel-soja

Comments

@aaronwmorris
Copy link
Contributor

I am receiving a segfault with pyindi-client on Ubuntu 24.04. It appears to happen when attempting to download the BLOB from the indiserver.

OS: Ubuntu 24.04
Platform: x86_64
INDI: 2.0.8
Python 3.12.3
libindi-dev 2.0.8+202406011046~ubuntu24.04.1 (latest from PPA as of this post)
pyindi-client @ git+https://github.com/indilib/pyindi-client.git@6f8fa8042f60f7b3079f6d66a0b8ee720af09d64

Debugging info:

$ gdb /home/aaron/git/indi-allsky/virtualenv/indi-allsky/bin/python3 /tmp/core._usr_bin_python3_12.1000.34c0e6c9-e161-4b1d-9e7c-9646008b9797.1119.28035
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/bin/python3...
(No debugging symbols found in /home/aaron/git/indi-allsky/virtualenv/indi-allsky/bin/python3)
[New LWP 1120]
[New LWP 1119]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python3 ./exposureTest.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000586be6 in PyObject_Malloc ()
[Current thread is 1 (Thread 0x70152ea006c0 (LWP 1120))]
(gdb) where
#0  0x0000000000586be6 in PyObject_Malloc ()
#1  0x0000000000584717 in _PyObject_New ()
#2  0x0000000000543acb in PyByteArray_FromStringAndSize ()
#3  0x000070152f4178f5 in _IBLOB_getblobdata (self=0x701528054b50) at indiclientpython_wrap.cpp:5891
#4  _wrap_IBLOB_getblobdata (self=<optimized out>, args=<optimized out>) at indiclientpython_wrap.cpp:17911
#5  0x0000000000581f32 in ?? ()
#6  0x0000000000549955 in PyObject_Vectorcall ()
#7  0x00000000005d7499 in _PyEval_EvalFrameDefault ()
#8  0x0000000000549d27 in ?? ()
#9  0x000000000054b5b3 in PyObject_CallMethodObjArgs ()
#10 0x000070152f47779f in SwigDirector_BaseClient::updateProperty (this=0xeb3fd0, property=...) at indiclientpython_wrap.cpp:3551
#11 0x000070152f4b3079 in INDI::BaseDevice::setValue(INDI::LilXmlElement const&, char*) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#12 0x000070152f4b5550 in INDI::WatchDeviceProperty::processXml(INDI::LilXmlElement const&, char*, std::function<INDI::ParentDevice ()> const&) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#13 0x000070152f4cd39f in INDI::AbstractBaseClientPrivate::dispatchCommand(INDI::LilXmlElement const&, char*) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#14 0x000070152f49d301 in INDI::BaseClientPrivate::BaseClientPrivate(INDI::BaseClient*)::{lambda(char const*, unsigned long)#1}::operator()(char const*, unsigned long) const ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#15 0x000070152f49d429 in std::_Function_handler<void (char const*, unsigned long), INDI::BaseClientPrivate::BaseClientPrivate(INDI::BaseClient*)::{lambda(char const*, unsigned long)#1}>::_M_invoke(std::_Any_data const&, char const*&&, unsigned long&&) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#16 0x000070152f4cee18 in TcpSocket::emitData(char const*, unsigned long) const ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#17 0x000070152f49c5de in INDI::TcpSocketSharedBlobs::readyRead() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#18 0x000070152f4cded2 in TcpSocketPrivate::processSocket() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#19 0x000070152f4d0004 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<TcpSocketPrivate::connectToHost(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned short)::{lambda(std::thread&&)#1}, std::thread> > >::_M_run() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312-x86_64-linux-gnu.so
#20 0x000070152eceabb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#21 0x000070152fa9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#22 0x000070152fb29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
@aaronwmorris
Copy link
Contributor Author

Here is the output from an execution of my test script. I ran this with the full python3-dbg packages

2024-06-25 11:28:50,960 [INFO] MainProcess shoot() [569]: Taking 15.00000000 s exposure (gain 1)
2024-06-25 11:28:51,063 [INFO] MainProcess run() [738]: Total time since last exposure 0.0534 s
2024-06-25 11:28:51,128 [INFO] MainProcess newMessage() [239]: new Message 2024-06-25T16:28:51: [ERROR] Got no stars, is gsc installed with appropriate environment variables set ??
2024-06-25 11:28:57,953 [INFO] MainProcess run() [684]: Camera last ready: 7.0s
2024-06-25 11:28:57,955 [INFO] MainProcess run() [685]: Exposure state: BUSY
2024-06-25 11:29:04,998 [INFO] MainProcess run() [684]: Camera last ready: 14.0s
2024-06-25 11:29:04,999 [INFO] MainProcess run() [685]: Exposure state: BUSY
2024-06-25 11:29:06,194 [INFO] MainProcess updateProperty() [185]: new Blob CCD1 for CCD Simulator
Fatal Python error: _PyMem_DebugMalloc: Python memory allocator called without holding the GIL
Python runtime state: initialized

Current thread 0x00007506af6006c0 (most recent call first):
  File "/home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/PyIndi.py", line 575 in getblobdata
  File "/home/aaron/git/indi-allsky/testing/exposureTest.py", line 232 in processBlob
  File "/home/aaron/git/indi-allsky/testing/exposureTest.py", line 186 in updateProperty

Thread 0x00007506b0b4c080 (most recent call first):
  File "/home/aaron/git/indi-allsky/testing/exposureTest.py", line 692 in run
  File "/home/aaron/git/indi-allsky/testing/exposureTest.py", line 769 in <module>

Extension modules: _PyIndi (total: 1)
Aborted (core dumped)

@aaronwmorris
Copy link
Contributor Author

Here is a full stack trace with the python debug symbols:

$ gdb /home/aaron/git/indi-allsky/virtualenv/indi-allsky/bin/python3-dbg /tmp/core._usr_bin_python3_12d.1000.34c0e6c9-e161-4b1d-9e7c-9646008b9797.4036.198881
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/bin/python3-dbg...
[New LWP 4037]
[New LWP 4036]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python3-dbg exposureTest.py'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

warning: 44     ./nptl/pthread_kill.c: No such file or directory
[Current thread is 1 (Thread 0x795dcc2006c0 (LWP 4037))]
(gdb) where
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000795dcd44526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000795dcd4288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x000000000060042a in fatal_error_exit (status=<optimized out>) at ../Python/pylifecycle.c:2735
#6  0x0000000000601baa in fatal_error (fd=2, header=header@entry=1, prefix=prefix@entry=0x785970 <__func__.5> "_PyMem_DebugMalloc",
    msg=msg@entry=0x74b7d0 "Python memory allocator called without holding the GIL", status=status@entry=-1) at ../Python/pylifecycle.c:2916
#7  0x0000000000601c1c in _Py_FatalErrorFunc (func=func@entry=0x785970 <__func__.5> "_PyMem_DebugMalloc",
    msg=msg@entry=0x74b7d0 "Python memory allocator called without holding the GIL") at ../Python/pylifecycle.c:2932
#8  0x000000000050681c in _PyMem_DebugCheckGIL (func=func@entry=0x785970 <__func__.5> "_PyMem_DebugMalloc") at ../Objects/obmalloc.c:2271
#9  0x0000000000506837 in _PyMem_DebugMalloc (ctx=0xb97e98 <_PyRuntime+312>, nbytes=56) at ../Objects/obmalloc.c:2280
#10 0x0000000000507b5e in PyObject_Malloc (size=<optimized out>) at ../Objects/obmalloc.c:801
#11 0x0000000000502ed9 in _PyObject_New (tp=tp@entry=0xa8cd80 <PyByteArray_Type>) at ../Objects/object.c:319
#12 0x00000000004a5bda in PyByteArray_FromStringAndSize (
    bytes=0x795dcb600010 "SIMPLE  =", ' ' <repeats 20 times>, "T / file does conform to FITS standard", ' ' <repeats 13 times>, "BITPIX  =", ' ' <repeats 19 times>, "16 / number of bits per data pixel", ' ' <repeats 18 times>, "NAXIS   =", ' ' <repeats 20 times>, "2 / number "..., size=2629440) at ../Objects/bytearrayobject.c:125
#13 0x0000795dccc9b57b in _IBLOB_getblobdata (self=self@entry=0x795dc4098d10) at indiclientpython_wrap.cpp:5891
#14 0x0000795dcccc8474 in _wrap_IBLOB_getblobdata (self=<optimized out>, args=args@entry=<WidgetViewBlob(this=<SwigPyObject at remote 0x795dcced3600>) at remote 0x795dcced3170>)
    at indiclientpython_wrap.cpp:17911
#15 0x00000000004ff0cb in cfunction_vectorcall_O (func=<built-in method IBLOB_getblobdata of module object at remote 0x795dcce8f4d0>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/methodobject.c:509
#16 0x00000000004b703d in _PyObject_VectorcallTstate (tstate=0x795dc40184a0,
    callable=callable@entry=<built-in method IBLOB_getblobdata of module object at remote 0x795dcce8f4d0>, args=args@entry=0x795dcd795190, nargsf=9223372036854775809,
    kwnames=kwnames@entry=0x0) at ../Include/internal/pycore_call.h:92
#17 0x00000000004b710d in PyObject_Vectorcall (callable=callable@entry=<built-in method IBLOB_getblobdata of module object at remote 0x795dcce8f4d0>,
    args=args@entry=0x795dcd795190, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/call.c:325
#18 0x00000000005aa7ef in _PyEval_EvalFrameDefault (tstate=0x795dc40184a0, frame=0x795dcd795130, throwflag=0) at Python/bytecodes.c:2706
#19 0x00000000005b0183 in _PyEval_EvalFrame (tstate=tstate@entry=0x795dc40184a0, frame=<optimized out>, throwflag=throwflag@entry=0) at ../Include/internal/pycore_ceval.h:89
#20 0x00000000005b02a2 in _PyEval_Vector (tstate=0x795dc40184a0, func=0x795dcd048290, locals=locals@entry=0x0, args=0x795dcc1f2e50, argcount=2, kwnames=0x0)
    at ../Python/ceval.c:1683
#21 0x00000000004b6d0f in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:419
#22 0x00000000004b703d in _PyObject_VectorcallTstate (tstate=tstate@entry=0x795dc40184a0, callable=callable@entry=<function at remote 0x795dcd048290>,
    args=args@entry=0x795dcc1f2e50, nargsf=nargsf@entry=2, kwnames=kwnames@entry=0x0) at ../Include/internal/pycore_call.h:92
#23 0x00000000004b7dd4 in object_vacall (tstate=tstate@entry=0x795dc40184a0,
    base=base@entry=<IndiClient(this=<SwigPyObject at remote 0x795dcced0360>, _timeout=<float at remote 0x795dcd31c9d0>, _exposure=15) at remote 0x795dcd023530>,
    callable=<function at remote 0x795dcd048290>, vargs=vargs@entry=0x795dcc1f2ec8) at ../Objects/call.c:850
#24 0x00000000004b7eeb in PyObject_CallMethodObjArgs (
    obj=<IndiClient(this=<SwigPyObject at remote 0x795dcced0360>, _timeout=<float at remote 0x795dcd31c9d0>, _exposure=15) at remote 0x795dcd023530>, name=<optimized out>)
    at ../Objects/call.c:911
#25 0x0000795dccd22880 in SwigDirector_BaseClient::updateProperty (this=0x1f9a8a0, property=...) at indiclientpython_wrap.cpp:6639
#26 0x0000795dccd59a91 in INDI::BaseDevice::setValue(INDI::LilXmlElement const&, char*) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#27 0x0000795dccd5bf68 in INDI::WatchDeviceProperty::processXml(INDI::LilXmlElement const&, char*, std::function<INDI::ParentDevice ()> const&) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#28 0x0000795dccd73837 in INDI::AbstractBaseClientPrivate::dispatchCommand(INDI::LilXmlElement const&, char*) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#29 0x0000795dccd43d95 in INDI::BaseClientPrivate::BaseClientPrivate(INDI::BaseClient*)::{lambda(char const*, unsigned long)#1}::operator()(char const*, unsigned long) const ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#30 0x0000795dccd43ebd in std::_Function_handler<void (char const*, unsigned long), INDI::BaseClientPrivate::BaseClientPrivate(INDI::BaseClient*)::{lambda(char const*, unsigned long)#1}>::_M_invoke(std::_Any_data const&, char const*&&, unsigned long&&) ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#31 0x0000795dccd752b0 in TcpSocket::emitData(char const*, unsigned long) const ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#32 0x0000795dccd43072 in INDI::TcpSocketSharedBlobs::readyRead() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#33 0x0000795dccd7436a in TcpSocketPrivate::processSocket() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#34 0x0000795dccd7649c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<TcpSocketPrivate::connectToHost(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned short)::{lambda(std::thread&&)#1}, std::thread> > >::_M_run() ()
   from /home/aaron/git/indi-allsky/virtualenv/indi-allsky/lib/python3.12/site-packages/_PyIndi.cpython-312d-x86_64-linux-gnu.so
#35 0x0000795dcc4eabb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#36 0x0000795dcd49ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#37 0x0000795dcd529c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

@aaronwmorris aaronwmorris mentioned this issue Aug 3, 2024
Closed
@aaronwmorris
Copy link
Contributor Author

Reconfirmed this behavior. I still experience the segfault with INDI 2.0.9.

@knro
Copy link
Contributor

knro commented Aug 4, 2024

@pawel-soja Do you think this was introduced with the changes in INDI v2.0.0 ?

@aaronwmorris
Copy link
Contributor Author

aaronwmorris commented Aug 4, 2024 •
edited
Loading

@knro @pawel-soja On a whim, I installed Python 3.11.9 on Ubuntu 24.04 Noble via the deadsnakes PPA and I did NOT experience the segfault. It sounds like this is a Python 3.12 and/or SWIG problem.

Edit: I also tested Python 3.12.4 on Ubuntu 22.04 Focal (via deadsnakes) and the segfault occurred. I just wanted to make sure this was not some new security feature of 24.04.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pawel-soja pawel-soja

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@knro @aaronwmorris @pawel-soja