create_column_filter error with base_url handling

[NiklasAbraham]

Hi, I was trying to build a cystoscape graph, for a package "pyeed" (bio-informatics) we are developing, and I think I found a bug.

I am working from a Docker Container (https://github.com/PyEED/PyEED_JupyterLab) with runs both jupyter-notebook and cytoscape in a compose,yml. This means I need to adjust the base_url for all my applications, and so fr everything worked. I can create the graph export it and so on.... But when creating a coulmn_filter (I want to hide all edges with a identity between a certain threshold and 1).
Code:

p4c.create_column_filter(
            'test',
            'identity',
            [threshold, 1.0],
            "BETWEEN",
            type="edges",
            apply=True,
            hide=True,
            base_url=base_url
        )

I ran into an error, the logs are the following:

DEBUG:py4...:Calling create_column_filter('test', 'identity', [0.85, 1.0], 'BETWEEN', type='edges', apply=True, 
hide=True, base_url='http://cytoscape:1234/v1')

DEBUG:py4...:ǀCalling set_current_network(None, base_url='http://cytoscape:1234/v1')

DEBUG:py4...:ǀǀCalling get_network_suid(None)

DEBUG:py4...:ǀǀǀCalling commands_post('network get attribute network="current" namespace="default" 
columnList="SUID"', base_url='http://127.0.0.1:1234/v1')

DEBUG:py4...:ǀǀǀHTTP POST(http://127.0.0.1:1234/v1/commands/network/get%20attribute), json: {'network': 'current', 
'namespace': 'default', 'columnList': 'SUID'}

INFO:backoff:Backing off _do_request_local(...) for 0.1s (requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: 
/v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7ffaafca1950>: Failed to establish a new connection: [Errno 111] Connection refused')))

DEBUG:py4...:ǀǀǀHTTP POST(http://127.0.0.1:1234/v1/commands/network/get%20attribute), json: {'network': 'current', 
'namespace': 'default', 'columnList': 'SUID'}

INFO:backoff:Backing off _do_request_local(...) for 1.0s (requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: 
/v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7ffab69635d0>: Failed to establish a new connection: [Errno 111] Connection refused')))

DEBUG:py4...:ǀǀǀHTTP POST(http://127.0.0.1:1234/v1/commands/network/get%20attribute), json: {'network': 'current', 
'namespace': 'default', 'columnList': 'SUID'}

ERROR:backoff:Giving up _do_request_local(...) after 3 tries (requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: 
/v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7ffab699d4d0>: Failed to establish a new connection: [Errno 111] Connection refused')))

In commands_post: HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffab699d4d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

DEBUG:py4...:ǀǀǀ'commands_post' exception ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', 
port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffab699d4d0>: Failed to establish a new 
connection: [Errno 111] Connection refused'))"))

DEBUG:py4...:ǀǀ'get_network_suid' exception ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', 
port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffab699d4d0>: Failed to establish a new 
connection: [Errno 111] Connection refused'))"))

DEBUG:py4...:ǀ'set_current_network' exception ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', 
port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffab699d4d0>: Failed to establish a new 
connection: [Errno 111] Connection refused'))"))

DEBUG:py4...:'create_column_filter' exception ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', 
port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffab699d4d0>: Failed to establish a new 
connection: [Errno 111] Connection refused'))"))

DEBUG:py4...:--------------------

For me this looks like it all starts well, and then when it is calling a command_post network_get_attribute it suddenly starts using 127.0.0.1 instead of the base_url I gave it which is http://cytoscape:1234/v1

This is a bug right?

[bdemchak]

Hi, Niklas ...

Indeed ... a terrible bug, and I have no idea why it hasn't been found before this. There are actually two bugs ... one in the get_network_suid() call in set_current_network(), and another in the check_supported_versions() call in create_column_filter().

Actually, I see more than just these two (ugh ... sorry).

I'll make the fixes, run a few regression tests, and then check them in. We don't have specific tests for the base_url parameters, so I'll await your report before closing this case.

Please stand by, and I'll let you know when there is a checkin for you to try.

[NiklasAbraham]

Okay that's perfect, thank you very much.

[bdemchak]

Hi, Niklas ...

I have completed the changes and checked them into branch 1.10.0 in Github:

https://github.com/cytoscape/py4cytoscape/tree/1.10.0

Are you able to check this branch out and try it with your system? If so, please let me know how it goes.

Thanks!

[NiklasAbraham]

Hi thank for the fast answer, yes I am able to test it, but sadly is still get this error.

---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/urllib3/connection.py:198, in HTTPConnection._new_conn(self)
    197 try:
--> 198     sock = connection.create_connection(
    199         (self._dns_host, self.port),
    200         self.timeout,
    201         source_address=self.source_address,
    202         socket_options=self.socket_options,
    203     )
    204 except socket.gaierror as e:

File /opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84 try:
---> 85     raise err
     86 finally:
     87     # Break explicitly a reference cycle

File /opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
     72     sock.bind(source_address)
---> 73 sock.connect(sa)
     74 # Break explicitly a reference cycle

ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

NewConnectionError                        Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py:793, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    792 # Make the request on the HTTPConnection object
--> 793 response = self._make_request(
    794     conn,
    795     method,
    796     url,
    797     timeout=timeout_obj,
    798     body=body,
    799     headers=headers,
    800     chunked=chunked,
    801     retries=retries,
    802     response_conn=response_conn,
    803     preload_content=preload_content,
    804     decode_content=decode_content,
    805     **response_kw,
    806 )
    808 # Everything went great!

File /opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py:496, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    495 try:
--> 496     conn.request(
    497         method,
    498         url,
    499         body=body,
    500         headers=headers,
    501         chunked=chunked,
    502         preload_content=preload_content,
    503         decode_content=decode_content,
    504         enforce_content_length=enforce_content_length,
    505     )
    507 # We are swallowing BrokenPipeError (errno.EPIPE) since the server is
    508 # legitimately able to close the connection after sending a valid response.
    509 # With this behaviour, the received response is still readable.

File /opt/conda/lib/python3.11/site-packages/urllib3/connection.py:400, in HTTPConnection.request(self, method, url, body, headers, chunked, preload_content, decode_content, enforce_content_length)
    399     self.putheader(header, value)
--> 400 self.endheaders()
    402 # If we're given a body we start sending that in chunks.

File /opt/conda/lib/python3.11/http/client.py:1281, in HTTPConnection.endheaders(self, message_body, encode_chunked)
   1280     raise CannotSendHeader()
-> 1281 self._send_output(message_body, encode_chunked=encode_chunked)

File /opt/conda/lib/python3.11/http/client.py:1041, in HTTPConnection._send_output(self, message_body, encode_chunked)
   1040 del self._buffer[:]
-> 1041 self.send(msg)
   1043 if message_body is not None:
   1044 
   1045     # create a consistent interface to message_body

File /opt/conda/lib/python3.11/http/client.py:979, in HTTPConnection.send(self, data)
    978 if self.auto_open:
--> 979     self.connect()
    980 else:

File /opt/conda/lib/python3.11/site-packages/urllib3/connection.py:238, in HTTPConnection.connect(self)
    237 def connect(self) -> None:
--> 238     self.sock = self._new_conn()
    239     if self._tunnel_host:
    240         # If we're tunneling it means we're connected to our proxy.

File /opt/conda/lib/python3.11/site-packages/urllib3/connection.py:213, in HTTPConnection._new_conn(self)
    212 except OSError as e:
--> 213     raise NewConnectionError(
    214         self, f"Failed to establish a new connection: {e}"
    215     ) from e
    217 # Audit hooks are only available in Python 3.8+

NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff103aac610>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/requests/adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    485 try:
--> 486     resp = conn.urlopen(
    487         method=request.method,
    488         url=url,
    489         body=request.body,
    490         headers=request.headers,
    491         redirect=False,
    492         assert_same_host=False,
    493         preload_content=False,
    494         decode_content=False,
    495         retries=self.max_retries,
    496         timeout=timeout,
    497         chunked=chunked,
    498     )
    500 except (ProtocolError, OSError) as err:

File /opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py:847, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    845     new_e = ProtocolError("Connection aborted.", new_e)
--> 847 retries = retries.increment(
    848     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    849 )
    850 retries.sleep()

File /opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py:515, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    514     reason = error or ResponseError(cause)
--> 515     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    517 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff103aac610>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[4], line 89
     77 """
     78 graph.set_layout(layout_name = "force-directed",
     79         properties_dict = {
   (...)
     85     )
     86 """
     87 p4c.unhide_all(base_url=base_url)
---> 89 p4c.create_column_filter(
     90             'test',
     91             'identity',
     92             [0.6, 1.0],
     93             "BETWEEN",
     94             type="edges",
     95             apply=True,
     96             hide=True,
     97             base_url=base_url,
     98         )
    100 p4c.notebook_export_show_image(filename="test", type="png", resolution=600, zoom=100.0, overwrite_file=True, base_url='http://cytoscape:1234/v1')

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:133, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    131     return log_return(func, value)
    132 except Exception as e:
--> 133     log_exception(func, e)
    134 finally:
    135     log_finally()

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:130, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    128 log_incoming(func, *args, **kwargs)
    129 try:
--> 130     value = func(*args, **kwargs) # Call function being logged
    131     return log_return(func, value)
    132 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/filters.py:188, in create_column_filter(filter_name, column, criterion, predicate, caseSensitive, anyMatch, type, hide, network, base_url, apply)
    184 cmd_json = {'id': 'ColumnFilter',
    185             'parameters': {'criterion': criterion, 'columnName': column, 'predicate': predicate,
    186                            'caseSensitive': caseSensitive, 'anyMatch': anyMatch, 'type': type}}
    187 cmd_body = {'name': filter_name, 'json': json.dumps(cmd_json)}
--> 188 return _create_filter_and_finish('commands/filter/create', cmd_body, hide, apply, network, base_url)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/filters.py:417, in _create_filter_and_finish(cmd, cmd_body, hide, apply, network, base_url)
    414     elif not apply:
    415         raise CyError('Attempt to create but not apply filter in Cytoscape version pre-3.9 is not supported')
--> 417 return _check_selected(hide, network, base_url)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/filters.py:435, in _check_selected(hide, network, base_url)
    433         res= style_bypasses.hide_nodes(network_selection.invert_node_selection(network=network, base_url=base_url)['nodes'])
    434     if sel_edges is not None and len(sel_edges) != 0:
--> 435         res = style_bypasses.hide_edges(network_selection.invert_edge_selection(network=network, base_url=base_url)['edges'])
    437 return {'nodes': sel_nodes, 'edges': sel_edges}

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:133, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    131     return log_return(func, value)
    132 except Exception as e:
--> 133     log_exception(func, e)
    134 finally:
    135     log_finally()

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:130, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    128 log_incoming(func, *args, **kwargs)
    129 try:
--> 130     value = func(*args, **kwargs) # Call function being logged
    131     return log_return(func, value)
    132 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/style_bypasses.py:2597, in hide_edges(edge_names, network, base_url)
   2549 @cy_log
   2550 def hide_edges(edge_names, network=None, base_url=DEFAULT_BASE_URL):
   2551     """Hide Edges.
   2552 
   2553     Hide (but do not delete) the specified edge or edges, by setting the Visible property bypass value to false.
   (...)
   2595         :meth:`set_edge_property_bypass`, :meth:`hide_edges`, :meth:`unhide_edges`, :meth:`unhide_all`
   2596     """
-> 2597     res = set_edge_property_bypass(edge_names, False, 'EDGE_VISIBLE', network=network, base_url=base_url)
   2598     return res

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:133, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    131     return log_return(func, value)
    132 except Exception as e:
--> 133     log_exception(func, e)
    134 finally:
    135     log_finally()

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:130, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    128 log_incoming(func, *args, **kwargs)
    129 try:
--> 130     value = func(*args, **kwargs) # Call function being logged
    131     return log_return(func, value)
    132 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/style_bypasses.py:300, in set_edge_property_bypass(edge_names, new_values, visual_property, bypass, network, base_url)
    246 @cy_log
    247 def set_edge_property_bypass(edge_names, new_values, visual_property, bypass=True, network=None,
    248                              base_url=DEFAULT_BASE_URL):
    249     """Set Edge Property Bypass.
    250 
    251     Set bypass values for any edge property of the specified edges, overriding default values and mappings defined by
   (...)
    298         :meth:`clear_edge_property_bypass`
    299     """
--> 300     net_suid = networks.get_network_suid(network, base_url=base_url)
    301     view_suid = network_views.get_network_views(net_suid, base_url=base_url)[0]
    302     edge_suids = edge_name_to_edge_suid(edge_names, network=network, base_url=base_url, unique_list=True)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:133, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    131     return log_return(func, value)
    132 except Exception as e:
--> 133     log_exception(func, e)
    134 finally:
    135     log_finally()

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:130, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    128 log_incoming(func, *args, **kwargs)
    129 try:
--> 130     value = func(*args, **kwargs) # Call function being logged
    131     return log_return(func, value)
    132 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/networks.py:268, in get_network_suid(title, base_url)
    266 # Make requested network current and return its SUID
    267 cmd = f'network get attribute network="{network_title}" namespace="default" columnList="SUID"'
--> 268 response = commands.commands_post(cmd, base_url=base_url)
    269 return int(response[0]['SUID'])

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:133, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    131     return log_return(func, value)
    132 except Exception as e:
--> 133     log_exception(func, e)
    134 finally:
    135     log_finally()

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/py4cytoscape_logger.py:130, in cy_log.<locals>.wrapper_log(*args, **kwargs)
    128 log_incoming(func, *args, **kwargs)
    129 try:
--> 130     value = func(*args, **kwargs) # Call function being logged
    131     return log_return(func, value)
    132 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/commands.py:397, in commands_post(cmd, base_url)
    395     return res['data']
    396 except requests.exceptions.RequestException as e:
--> 397     _handle_error(e)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/commands.py:390, in commands_post(cmd, base_url)
    388 post_body = _command_2_post_query_body(cmd)
    389 headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}
--> 390 r = _do_request('POST', post_url, json=post_body, headers=headers, base_url=base_url)
    391 r.raise_for_status()
    392 res = json.loads(r.text)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/commands.py:700, in _do_request(method, url, base_url, raw_request, **kwargs)
    697 if not raw_request:
    698     do_initialize_sandbox(requester, base_url=base_url) # make sure there's a sandbox before executing a command
--> 700 return requester(method, url, **kwargs)

File /opt/conda/lib/python3.11/site-packages/backoff/_sync.py:105, in retry_exception.<locals>.retry(*args, **kwargs)
     96 details = {
     97     "target": target,
     98     "args": args,
   (...)
    101     "elapsed": elapsed,
    102 }
    104 try:
--> 105     ret = target(*args, **kwargs)
    106 except exception as e:
    107     max_tries_exceeded = (tries == max_tries_value)

File /opt/conda/lib/python3.11/site-packages/py4cytoscape/commands.py:689, in _do_request_local(method, url, **kwargs)
    685 @backoff.on_exception(backoff.expo, requests.exceptions.ConnectionError, max_tries=10)
    686 def _do_request_local(method, url, **kwargs):
    687     # Call CyREST via a local URL
    688     log_http_request(method, url, **kwargs)
--> 689     r = requests.request(method, url, **kwargs)
    690     log_http_result(r)
    691     return r

File /opt/conda/lib/python3.11/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File /opt/conda/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    584 send_kwargs = {
    585     "timeout": timeout,
    586     "allow_redirects": allow_redirects,
    587 }
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File /opt/conda/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    700 start = preferred_clock()
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)
    706 elapsed = preferred_clock() - start

File /opt/conda/lib/python3.11/site-packages/requests/adapters.py:519, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    515     if isinstance(e.reason, _SSLError):
    516         # This branch is for urllib3 v1.22 and later.
    517         raise SSLError(e, request=request)
--> 519     raise ConnectionError(e, request=request)
    521 except ClosedPoolError as e:
    522     raise ConnectionError(e, request=request)

ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=1234): Max retries exceeded with url: /v1/commands/network/get%20attribute (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff103aac610>: Failed to establish a new connection: [Errno 111] Connection refused'))

[bdemchak]

Actually, Niklas ... it would be very valuable to have the py4cytoscape.log file ... or even the last several hundred lines. It's in logs/py4cytoscape.log. This log gives a blow by blow of function calls, parameters and return values. It should show exactly where the base_url is getting lost. Would you mind sending that?