Quattrocchi (four eyes, or person that wears glasses in Italian) is a mixed model, event and frame based, to infer disparity from a couple of DAVIS Neuromorphic Cameras. This work purpose is to exploit the possibilities offered by these sensors, defining an algorithm that could be used in real world applications that need energy efficient solutions, while also trying to infer principles of computation in the mammalian brain, thanks to the possibilities offered by neuromorphic solutions.
The project is based on the assumption of cooperation between the Magno and Parvo visual pathways. The Magno scripts are written in Python and simulate a spiking neural network meant to work, in real time, on Dynap-se (or other neuromorphic devices), a neuromorphic processor developed at the Neuroinformatic institute of Zurich. The Parvo scripts are in Matlab and they implement a rate based network encoding disparity from stereo frames, and a first concept of the processes supposedly connecting the two streams. A traditional multiscale energy model is also implemented for comparison. A different Magno_Pathway implementation is in the work and it will be in Opencl, in order to build a small and energy efficient solution in the future, for stereo active vision.
The Magno_Pathway is supposed to be used on Spyder, packet managing with anaconda is also advised. The dependency are listed here:
-
Brian2 (The framework for SNN simulation)
-
Teili (A Library for dynap-se simulation based on Brian2) (The repo is still private and it will be opened to the public soon)
and also:
- numpy
- scipy
- pypng
- pyqtgraph
- pyopengl
The Data folders should house all the data needed for the models to work, and files generated by the scripts. Their contents won't be uploaded on the repo
The Magno Path has the excecution priority since it is computing the coarse disparity estimation. The normal flow is to load the recordings in the Data folder, compute them through Monocular_neurons.py then Binocular_neurons.py, and extract the frames with Magno_Pathway/Tools/AedatMovieMaker.py The frames need to be converted in .mat with Parvo_Pathway/Tools/FramesImporter.m And then it will possible to run /Parvo_Pathway/Disparity_Surfaces.m to compute the Parvo network activity, merged with the result of the Magno network. The final results can be confronted with the ones from the script /Parvo_Pathway/Energy_Model_Pyramidal.m