Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accelerate image processing on the client with GPU.js #294

Merged
merged 58 commits into from
Jan 12, 2022
Merged

Conversation

tddough98
Copy link
Collaborator

Here, we improve the image processing and display for the DeepCell Label client with GPU acceleration. I revamped existing ____Canvascomponents to use a kernel compiled with GPU.js instead of processing image data on the CPU, resulting in large speedups for image processing, especially for larger images. For a 4.5 megapixel image, changing the range of a channel now take around 6ms to complete the render compared to about 80ms in our existing deployment.

Each Canvas is now explicitly composed into a single canvas, instead of implicitly composed with a stack of canvases in the same position on the page. This address issue #274.

I've also addressed #273 by sending some additional ADD_LAYER and REMOVE_LAYER events to parent actors to ensure that the relevant hooks update when the event occurs.

I removed routes and logic from the backend that are now exclusively handled by the client including

  • creating a label image from the label array with a colormap
  • creating outlines from the label array

@willgraf willgraf added the enhancement New feature or request label Jan 7, 2022
Copy link
Contributor

@willgraf willgraf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I think this is a good opportunity to add some tests to the Canvas components, especially as we have moved some more complicated logic into them.

It looks as though there is a failure on the 17.x node version for tests as well.

deepcell_label/blueprints.py Outdated Show resolved Hide resolved
deepcell_label/imgutils.py Outdated Show resolved Hide resolved
deepcell_label/models.py Outdated Show resolved Hide resolved
visualizer/src/Canvas/Labeled/LabeledCanvas.js Outdated Show resolved Hide resolved
visualizer/src/Canvas/Labeled/LabeledCanvas.js Outdated Show resolved Hide resolved
@willgraf willgraf self-requested a review January 11, 2022 21:25
Copy link
Contributor

@willgraf willgraf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Some unit tests for the individual canvas components would be nice to include though.

@tddough98
Copy link
Collaborator Author

I added a test that Canvas sends events to the canvasMachine and that BrushCanvas has a canvas of the right size. I spent some time trying to test the canvas contents of BrushCanvas, but I hit some walls. As Jest runs tests in a node environment, some browser APIs, including the canvas, are not available. I installed canvas as a dev dependency which recreates the canvas API for node, but this does not include webgl functionality, which is a key need here. One solution seemed to be running tests with jest-electron so that webgl is available, but that will require ejecting create-react-app, which I'm not inclined to do in this PR. I'll explore that more another time, but for now, the two components tested should provide at least some foundation and models for testing more canvas functionality in the future.

@tddough98
Copy link
Collaborator Author

Unfortunately another bamboozling update for testing components with GPU.js. The standard tool for javascript code coverage is istanbul, which turns out is incompatible with GPU.js gpujs/gpu.js#663. The recommendation is to use v8 code coverage instead, but this requires using the coverageProvider option for jest, which is not supported with create-react-app...

I'm taking out the BrushCanvas test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Label highlighting in chrome covers the raw image Fix canvas updates when adding or removing channels
2 participants