Queue is incompatible with macOS - on ARM architecture

[tonybaloney]

I get this error in macOS when creating a queue. SO threads seem to suggest this happens when creating in a post-fork process, but it's all occurring within the constructor chain

[2023-05-25 12:18:29 +1000] [40612] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/Users/anthonyshaw/projects/cpython/.venv-312/lib/python3.12/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/Users/anthonyshaw/projects/cpython/experiment/subinterpreter_async_worker.py", line 80, in init_process
    self.tpool = self.get_thread_pool()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/anthonyshaw/projects/cpython/experiment/subinterpreter_async_worker.py", line 87, in get_thread_pool
    return SubinterpreterPoolExecutor(max_workers=self.cfg.threads)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/anthonyshaw/projects/extrainterpreters/src/extrainterpreters/executor.py", line 49, in __init__
    self._work_queue = Queue()
                       ^^^^^^^
  File "/Users/anthonyshaw/projects/extrainterpreters/src/extrainterpreters/queue.py", line 405, in __init__
    self._signal_pipe = _SimplexPipe()
                        ^^^^^^^^^^^^^^
  File "/Users/anthonyshaw/projects/extrainterpreters/src/extrainterpreters/queue.py", line 118, in __init__
    self._post_init()
  File "/Users/anthonyshaw/projects/extrainterpreters/src/extrainterpreters/queue.py", line 148, in _post_init
    EISelector.register(self.reader_fd, selectors.EVENT_READ, self._read_ready_callback)
  File "/Users/anthonyshaw/projects/extrainterpreters/src/extrainterpreters/resources.py", line 22, in register
    self.selector.register(file, event, (callback,))
  File "/Users/anthonyshaw/projects/cpython/Lib/selectors.py", line 522, in register
    self._selector.control([kev], 0, 0)
OSError: [Errno 9] Bad file descriptor

[jsbueno]

with luck this could be a bug in my code - after all, file descriptors should be valid process wide.

It does not seem I can get an easy shell on MacOS - but maybe, if it is not a problem in my code, the issue can be work around by picking an specific Selector in the stdlibs selector module (I just went with "DefaultSelector".

[jsbueno]

Hi! I got access to a MacOS shell -
The tests passed, and I could, in the interactive interpreter, pickle a Queue into a subinterpreter and communicate using both ends.

Did you get the error simply by trying to instantiate a Queue on the main interpreter?

There are some guards to make it harder for people crashing the Python runtime which checks from where certain functions where called, the "extrainterpreters" module being white-listed. But these are faulty right now(I will likely remove them altogether, as there are saner safety mechanisms now) - but just in case, set a global variable "DEBUG=True" in the main package before doing whatever trigger your error above!

(I will perform futher tests on the Mac later on)

[network-shark]

I tried it on my MacBook M1 and pytest is segfaulting .

Let me know how I can help.

╰─ pytest -vvv
================================================================================================================================= test session starts =================================================================================================================================
platform darwin -- Python 3.12.0, pytest-7.4.2, pluggy-1.3.0 -- /Users/sharky/.pyenv/versions/3.12.0/envs/subinterpreter/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/sharky/python/downloads/extrainterpreters
configfile: pytest.ini
collected 54 items

test_basic.py::test_running_plain_call_works PASSED                                                                                                                                                                                                                             [  1%]
test_basic.py::test_interpreter_is_destroyed_after_context_exits PASSED                                                                                                                                                                                                         [  3%]
test_basic.py::test_extrainterpreters_list_all PASSED                                                                                                                                                                                                                           [  5%]
test_basic.py::test_interpreter_closes_out_of_scope PASSED                                                                                                                                                                                                                      [  7%]
test_basic.py::test_interpreter_cant_be_started_twice PASSED                                                                                                                                                                                                                    [  9%]
test_basic.py::test_running_threaded_call_works PASSED                                                                                                                                                                                                                          [ 11%]
test_basic.py::test_running_threaded_call_works_local PASSED                                                                                                                                                                                                                    [ 12%]
test_basic.py::test_interpreter_is_running SKIPPED (Closing the interpreter fails if 'is_running' is called while it is running. Wait for fix in interpreters or workaround)                                                                                                    [ 14%]
test_basic.py::test_interpreter_fails_trying_to_send_data_larger_than_buffer PASSED                                                                                                                                                                                             [ 16%]
test_basic.py::test_interpreter_fails_trying_to_receive_data_larger_than_buffer PASSED                                                                                                                                                                                          [ 18%]
test_basic.py::test_interpreter_target_argument PASSED                                                                                                                                                                                                                          [ 20%]
test_basic.py::test_interpreter_run_can_instantiate_class PASSED                                                                                                                                                                                                                [ 22%]
test_basic.py::test_interpreter_run_can_call_classmethod PASSED                                                                                                                                                                                                                 [ 24%]
test_basic.py::test_interpreter_run_can_call_callable PASSED                                                                                                                                                                                                                    [ 25%]
test_boards.py::test_lockableboard_single_item_back_and_forth PASSED                                                                                                                                                                                                            [ 27%]
test_boards.py::test_lockableboard_to_other_interpreter PASSED                                                                                                                                                                                                                  [ 29%]
test_boards.py::test_structlock_works PASSED                                                                                                                                                                                                                                    [ 31%]
test_boards.py::test_structlock_reentrant PASSED                                                                                                                                                                                                                                [ 33%]
test_boards.py::test_structlock_locks PASSED                                                                                                                                                                                                                                    [ 35%]
test_boards.py::test_structlock_timeout_is_restored_on_acquire_fail[None] PASSED                                                                                                                                                                                                [ 37%]
test_boards.py::test_structlock_timeout_is_restored_on_acquire_fail[0.0001] PASSED                                                                                                                                                                                              [ 38%]
test_boards.py::test_remotearray_base PASSED                                                                                                                                                                                                                                    [ 40%]
test_boards.py::test_remotearray_not_deleted_before_ttl_expires PASSED                                                                                                                                                                                                          [ 42%]
test_boards.py::test_memoryboard_fetch_from_gone_interpreter_doesnot_crash PASSED                                                                                                                                                                                               [ 44%]
test_low_level.py::test_lowlevel_func_guarded_from_out_of_package_call[_address_and_size] PASSED                                                                                                                                                                                [ 46%]
test_low_level.py::test_lowlevel_func_guarded_from_out_of_package_call[_remote_memory] PASSED                                                                                                                                                                                   [ 48%]
test_low_level.py::test_lowlevel_func_guarded_from_out_of_package_call[_atomic_byte_lock] PASSED                                                                                                                                                                                [ 50%]
test_low_level.py::test_get_address_works PASSED                                                                                                                                                                                                                                [ 51%]
test_low_level.py::test_remote_memory_works PASSED                                                                                                                                                                                                                              [ 53%]
test_low_level.py::test_remote_memory_works_across_interpreters PASSED                                                                                                                                                                                                          [ 55%]
test_low_level.py::test_atomiclock_works_only_when_indicator_at_0 PASSED                                                                                                                                                                                                        [ 57%]
test_low_level.py::test_atomiclock_locks PASSED                                                                                                                                                                                                                                 [ 59%]
test_queue.py::test_simplexpipe_works PASSED                                                                                                                                                                                                                                    [ 61%]
test_queue.py::test_simplexpipe_unpickles_with_same_memory_buffer_main_intrepreter PASSED                                                                                                                                                                                       [ 62%]
test_queue.py::test_simplexpipe_unpickles_with_same_memory_buffer_child_intrepreter PASSED                                                                                                                                                                                      [ 64%]
test_queue.py::test_simplexpipe_file_registers_unregister_from_global_selector PASSED                                                                                                                                                                                           [ 66%]
test_queue.py::test_simplexpipe_closes_fds_on_exit PASSED                                                                                                                                                                                                                       [ 68%]
test_queue.py::test_simplexpipe_doesnotclose_when_open_in_other_interpreter PASSED                                                                                                                                                                                              [ 70%]
test_queue.py::test_duplexpipe_is_unpickled_as_counterpart_and_comunicates PASSED                                                                                                                                                                                               [ 72%]
test_queue.py::test_locackle_duplexpipe_locks PASSED                                                                                                                                                                                                                            [ 74%]
test_queue.py::test_singlequeue_is_unpickled_as_counterpart_and_comunicates PASSED                                                                                                                                                                                              [ 75%]
test_queue.py::test_queue_send_object PASSED                                                                                                                                                                                                                                    [ 77%]
test_queue.py::test_queue_sent_to_other_interpreter PASSED                                                                                                                                                                                                                      [ 79%]
test_queue.py::test_queue_each_value_is_read_in_a_single_interpreter PASSED                                                                                                                                                                                                     [ 81%]
test_queue.py::test_queue_get_value_from_subinterpreter PASSED                                                                                                                                                                                                                  [ 83%]
test_queue.py::test_queue_trying_to_get_value_from_closed_interpreter_doesnot_break_queue PASSED                                                                                                                                                                                [ 85%]
test_queue.py::test_queue_subinterpreters_can_exchange_data PASSED                                                                                                                                                                                                              [ 87%]
test_struct.py::test_struct_size PASSED                                                                                                                                                                                                                                         [ 88%]
test_struct.py::test_struct_works PASSED                                                                                                                                                                                                                                        [ 90%]
test_struct.py::test_struct_with_offset PASSED                                                                                                                                                                                                                                  [ 92%]
test_struct.py::test_struct_detach PASSED                                                                                                                                                                                                                                       [ 94%]
test_struct.py::test_struct_from_values PASSED                                                                                                                                                                                                                                  [ 96%]
test_struct.py::test_struct_bytes PASSED                                                                                                                                                                                                                                        [ 98%]
test_struct.py::test_struct_doublefield PASSED                                                                                                                                                                                                                                  [100%]

============================================================================================================================ 53 passed, 1 skipped in 4.73s ============================================================================================================================
Exception ignored in: <function _PipeBase.__del__ at 0x102dc6e80>
Traceback (most recent call last):
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/queue.py", line 130, in __del__
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/queue.py", line 104, in close
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/memoryboard.py", line 81, in __enter__
TimeoutError: Timeout trying to acquire lock
Exception ignored in: <function RemoteArray.__del__ at 0x102dc2ca0>
Traceback (most recent call last):
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/memoryboard.py", line 418, in __del__
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/memoryboard.py", line 384, in close
  File "/Users/sharky/python/downloads/extrainterpreters/src/extrainterpreters/memoryboard.py", line 81, in __enter__
TimeoutError: Timeout trying to acquire lock
python3.12(40213,0x1eb125e00) malloc: *** error for object 0x102fe81b0: pointer being freed was not allocated
python3.12(40213,0x1eb125e00) malloc: *** set a breakpoint in malloc_error_break to debug

[jsbueno]

I think this traceback can help already - it looks like we have a double dealocation there.