Add in Qt5/Qwt5 changes from Tony

[bennahugo]

Starting an integration test for this branch of the backend. I have spit and merged tony's changes into Timba and into MeqGUI.
Co-authored-by: Tony Willis

[bennahugo]

@twillis449 which plots are currently not working? I'm running against the latest version of pyqwt5 although. I have the following as dependencies:

 71 install_requires = [
 72     'numpy',
 73     'python-casacore',
 74     'six',
 75     'configparser',
 76     'matplotlib',
 77     'PythonQwt>=0.10.1'
 78 ] + [ # meqtrees sister packages    
 79     'purr',
 80     'astro-kittens',
 81     'meqtrees-cattery',
 82     'astro-tigger-lsm',
 83     'owlcat',
 84 ]

along with a dependency on Qt4 which can now be changed to pyqt 5? I can probably remove the sip croft in meqbrowser now to drop the implicit dependency on sip

[bennahugo]

So I believe we now depend on https://pypi.org/project/QtPy/ ? I will need to add that to setup.py. You don't need sip do you?

[bennahugo]

Not passing yet

2022/03/08 14:47:13 INFO: all commands have executed successfully
2022/03/08 14:47:13 INFO: elapsed time is 0h0m2.15s
drwxr-sr-x 15 root staff 4096 Mar  8 14:47 /usr/local/lib/python3.8/dist-packages/Pyxis/recipes/meqtrees-batch-test/WSRT.MS
Successful read/write open of default-locked table /usr/local/lib/python3.8/dist-packages/Pyxis/recipes/meqtrees-batch-test/WSRT.MS: 26 columns, 7560 rows
�[91mTraceback (most recent call last):
  File "/usr/local/bin/downweigh-redundant-baselines.py", line 105, in <module>
    from Cattery.Meow import IfrSet
  File "/usr/local/lib/python3.8/dist-packages/Cattery/Meow/__init__.py", line 50, in <module>
    from .CorruptComponent import CorruptComponent
  File "/usr/local/lib/python3.8/dist-packages/Cattery/Meow/CorruptComponent.py", line 29, in <module>
    from Timba.TDL import *
  File "/usr/local/lib/python3.8/dist-packages/Timba/TDL/__init__.py", line 32, in <module>
    from Timba.TDL.TDLOptions import TDLOption,TDLMenu,TDLJob,TDLOptionSeparator
  File "/usr/local/lib/python3.8/dist-packages/Timba/TDL/TDLOptions.py", line 69, in <module>
    class OptionConfigParser (configparser.RawConfigParser):
  File "/usr/local/lib/python3.8/dist-packages/Timba/TDL/TDLOptions.py", line 71, in OptionConfigParser
    mandatoryOptionsSet = Signal()
NameError: name 'Signal' is not defined
�[0m�[91mE
======================================================================
ERROR: batch_test.testMeqtreesBatchJob
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/src/pyxis-1.7.4.3/Pyxis/recipes/meqtrees-batch-test/batch_test.py", line 109, in testMeqtreesBatchJob
    run("{0:s} downweigh-redundant-baselines {1:s}/WSRT.MS".format(owlcat, PACKAGE_TEST_DIR));
  File "/src/pyxis-1.7.4.3/Pyxis/recipes/meqtrees-batch-test/batch_test.py", line 43, in run
    raise RuntimeError("failed with exit code %x"%code);
RuntimeError: failed with exit code 1

[twillis449]

I don't understand why you have a dependency for matplotlib. At least, 'back in the day' qwt was chosen over matplotlib because qwt was much faster than matplotlib. I think all the plotting stuff uses PyQt5 and, in that sense, should work reasonably well. However, while the visibility plotting works OK with ATCA or 'old' pre-Apertif WSRT data sets, there's no way it's going to handle current MeerKAT visibility data sets - - well, maybe things will work on some giant computer cluster, but not on my laptop - LOL. For my amusement, I requested a UV data set for one of the MeekKAT cluster legacy survey fields, and the archive said it was going to send me a 3 Tb file! Since my laptop only has 1.5 Tb storage, I had to decline!

I don't use sip directly, so I don't think you have to worry about that.

[ludwigschwardt]

I don't understand why you have a dependency for matplotlib.

I did a quick search of the code and it looks like it's not imported anywhere so probably safe to remove.

For my amusement, I requested a UV data set for one of the MeekKAT cluster legacy survey fields, and the archive said it was going to send me a 3 Tb file! Since my laptop only has 1.5 Tb storage, I had to decline!

Yip! On a laptop your best option is to use katdal natively and pull the data a few chunks at a time for inspection. But then you still have to figure out the visualisation part...

[bennahugo]

Thanks @twillis449. Yes I believe we can remove matplotlib from meqtrees-frontend's requirement list. It is in use (along with bokeh) by owlcat, so it will be pulled in through that requirements list.

I'm still working on polishing this so that the backend works in the absence of qt5 (pipeline mode), but I should be able to wrap that up today. I will do some testing on the frontend to make sure everything, including install, works as expected before opening a PR for that

[bennahugo]

Hmm no... it seems the new changes in signaling is breaking the standalone QObject in PyApps/Apps/QObject.py making the pipeliner impossible to use when Qt is not installed. My Qt knowledge is super limited here so not sure what a good fix would be @twillis449

[bennahugo]

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/Timba/octopussy.py", line 490, in run
    self._dispatch_whenevers(msg);
  File "/usr/local/lib/python3.8/dist-packages/Timba/octopussy.py", line 282, in _dispatch_whenevers
    we.fire(msg);
  File "/usr/local/lib/python3.8/dist-packages/Timba/octopussy.py", line 148, in fire
    return self.target(*self.args,**self.kwargs);
  File "/usr/local/lib/python3.8/dist-packages/Timba/Apps/multiapp_proxy.py", line 330, in _hello_handler
    server = self._connect_server(addr);
  File "/usr/local/lib/python3.8/dist-packages/Timba/Apps/multiapp_proxy.py", line 300, in _connect_server
    self.client.serverConnected.emit(server)
AttributeError: 'Client' object has no attribute 'serverConnected'

[bennahugo]

Hmm ... should this not read self.serverConnected (multiapp_proxy(QObject) has a bunch of signals at class scope)

[bennahugo]

Nope solidly stuck at this point - reverting my changes. It looks like the entire signalling standing will need to be rewritten for this to work

[twillis449]

Hum - my only suggestion ... there are some other python (and also C++) packages that aim to provide the signal/slot mechanism without PyQt (or QObjct) or Qt. Have a look at https://github.com/dgovil/PySignal and references given at the bottom there. Possibly something can be adapted from there.

[bennahugo]

Thanks, I'm still working on a fix for this

[bennahugo]

@twillis449 I merged in your gui changes on meqtrees-frontend. However I see there is a ton of places where you say it does not work for Python3. Is there something wrong with the qt package? Do we need to rebuild it from source for another version?

Edit: I'm still working on implementing an alternative signalling system for the backend. Unfortunately the package mentioned does not really provide QObject support so I'm going to go with something not-so-Qt specific as different codepaths when running headless

[bennahugo]

Also @twillis449 have you already made changes to Kittens and Purr? I notice kittens are still being pulled in and that still has qt4 dependencies. I can make changes to the signals, but I want to check with you first