[BUG] Magics stop displaying output after several executions

[michaelnchin]

Community Note

• Please use a 👍 reaction to provide a +1/vote. This helps the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Describe the bug

A user running graph-notebook on their local Python 3.9 environment experienced an issue where Notebook cells stop responding for the same query after few executions, even for very simple SPARQL queries like:

MATCH (u:User)
RETURN u.uuid

For the first few executions, the query runs fine, and the results are displayed in the notebook. However, future executions of the same query eventually fail to display anything. The query appears to not be stuck since the cell gets completed, but no response is shown.

The Jupyter logs from executing the problematic magic query show the following errors:

E 22:12:29.671 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x110a8cca0>)
    Traceback (most recent call last):
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/ioloop.py", line 738, in _run_callback
        ret = callback()
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 718, in <lambda>
        self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
        self._handle_recv()
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/-neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 334, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 1081, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 1056, in _write_frame
        return self.stream.write(frame)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 539, in write
        self._handle_write()
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 965, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[E 22:12:29.656 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 334, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 1081, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/websocket.py", line 1056, in _write_frame
        return self.stream.write(frame)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 539, in write
        self._handle_write()
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 965, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/-/neptune-graph-notebook/lib/python3.9/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError

[michaelnchin]

The observed error matches the signature of a recent issue related to dependency version conflicts inside the Jupyter core notebook package: jupyter/notebook#6721

The notebook v6.5.5 patch release includes a fix for this issue, pinning jupyter_client<8 and ipyzmq<25.

[michaelnchin]

The patched version that would pin jupyter_client<8 and ipyzmq<25 is not being pulled in because we currently pin graph-notebook to use notebook<6.5.0. The original reason we added this pin was to resolve an issue specific to 6.5.1; I assume no risk in upgrading to the latest version of Notebook 6.x.

To mitigate this, we should update setup.py and requirements.txt to raise the ceiling of the notebook dependency to <7.0.0.