atem-connection not playing well with webpack and electron. #106

sensslen · 2021-03-20T21:45:19Z

I'm submitting a ...
[ x ] bug report
[ ] feature request
[ ] question about the decisions made in the repository
[ ] question about how to use this project
Summary
I am trying to use atem-connection in an electron application. Electron-Forge (the electron helper tool used) normally uses webpack in order to pack the application (and allow for usage of react.

When I use atem-connection on the main process, threadedClass fails to find the file atemSocketChild, as it searches the current directory.

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. StackOverflow, personal fork, etc.)

To reproduce the issue feel free to check out https://github.com/sensslen/Cgf.CameraControl.Main.Gui

Julusian · 2021-03-20T23:01:38Z

Could you try this version on npm 2.3.0-nightly-wip-update-threadedclass-20210313-231347-ec8d035.0.
I think it should resolve your issues

sensslen · 2021-03-21T08:23:53Z

Thanks a lot for your answer. Unfortunately i does not. It seems that threadedClass still looks for a file instead of querying webpack for the file to load.

This is the Error I get:

(node:8800) UnhandledPromiseRejectionWarning: Error: Cannot find module 'D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\atemSocketChild'
Require stack:
- D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js
- D:\GIT\Private\Cgf.CameraControl.Main.Gui\node_modules\electron\dist\resources\default_app.asar\main.js
-
    at Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.n._resolveFilename (electron/js2c/browser_init.js:261:1128)
    at Function.resolve (internal/modules/cjs/helpers.js:94:19)
    at D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:120749:40
    at new Promise (<anonymous>)
    at Object.threadedClass (D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:120640:12)
    at AtemSocket.<anonymous> (D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:17384:57)
    at Generator.next (<anonymous>)
    at D:\GIT\Private\Cgf.CameraControl.Main.Gui\.webpack\main\index.js:121484:71
    at new Promise (<anonymous>)

Julusian · 2021-03-21T12:00:57Z

oh sorry, I glossed over that you are using webpack.

Unfortunately I don't have any ideas on how to resolve this. I think the problem here is that it is trying to create a new worker thread, and the worker thread is failing to load its entrypoint. And even then the dynamic imports will probably have messed up the bundling so that the intended code will be missing. Webpack is not something I am comfortable with so do not know where to start

sensslen · 2021-03-21T18:28:38Z

No worries, neither do I have much knowledge about webpack.

would you mind telling me what purpose threadedclass serves so that I may try to spin a branch without it and thus make things work even with webpack?

Julusian · 2021-03-21T19:26:00Z

threadedclass is used to manage a worker thread. I am currently tempted to see if it is possible to swap out the library for https://github.com/andywer/threads.js, as that is more widely used and claims to work well with webpack.

The thread is necessary for handling the connection, to avoid the connection dropping when the node event loop is blocked by other tasks.

sensslen · 2021-03-21T20:49:47Z

Let me give this a try.

sensslen · 2021-03-23T06:30:38Z

Just to keep you up to date with my efforts. I'm currently trying to write a webpack plugin to handle threadedClass... This seems less intrusive than changing the used library.

sensslen · 2021-03-23T20:47:45Z

While workin on this issue, I found that webpack is not as easy to master as I initially thought. You mention the node event loop being blocked. Did you encounter this? Would it be feasable to spin a variant of atem-connection that runs everything on the event loop - I'd be willing to create and maintain this spin off...

Julusian · 2021-03-23T20:58:21Z

Yeah, I wouldnt want to dive into that side of webpack.
Yes we found this very easy to hit. Even things as simple as parsing a chunk of json, or just while loops that dont yield.

If your code is using enough timers, or waiting on promises then it might be ok. But any synchronous operations that take more than 20ms are likely to cause issues.

sensslen · 2021-03-28T19:19:57Z

I found a way around this issue. There is the possibility to specify an external source for webpack (to tell it that this particular import should not be grabbed). This allows me to mark the atemSocketChild class as external. I the use WebpackCopyPlugin to copy the file over to the destination directory. With this trick I was able to get things to work properly. Thanks for the insights.

Maybe you want to add this to the documentation.

https://webpack.js.org/plugins/copy-webpack-plugin/ :

   new CopyPlugin({
        patterns: [{ from: '**/atemSocketChild*', to: '[name].[ext]' }],
    }),

(Note that the newest version does only support webpack 5)

https://v4.webpack.js.org/configuration/externals/ :

externals: /^(atemSocketChild)$/i,

chrisspiegl · 2021-05-14T10:55:06Z

I just ran into something that feels related to this.
I am trying to build an application that has a Electron Menu Bar interface and using electron-builder I get this error message:

Worker Thread error Error: Cannot find module '/Users/*******/midi2atem/dist/mac/MIDI2ATEM.app/Contents/Resources/app.asar/node_modules/threadedclass/dist/child-process/threadedclass-worker.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.Module._load (internal/modules/cjs/loader.js:732:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:174:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Could this be related? And is there a possible workaround?

sensslen · 2021-05-14T15:19:23Z

This is definitively related. I was able to solve the issue by creating a second target for webpack that packages atemSocketChild class and it's dependencies. That way it works flawlessly. Please feel free to take https://github.com/sensslen/Cgf.CameraControl.Main.Gui (https://github.com/sensslen/Cgf.CameraControl.Main.Gui/blob/main/webpack.main.config.js) for inspiration.

chrisspiegl · 2021-05-19T10:53:15Z

@sensslen Thank you for sharing your config. At the moment I am probably just going to stick to unbuild versions… introducing webpack additionally was not on my plan for this moment.

Still hopeful that this lib could just support building out of the box.

Julusian · 2021-05-19T12:42:18Z

Im going to reopen this as a reminder to myself to look into what can be done here.

It's a tough one to handle, as webpack introduces some complexity (and I suspect that the config will need to in some way be aware of the need to do some extra work), and electron introduces some other complexity due to its bad support for worker_threads.

I have been tempted to try switching to a different thread helper library, which does itself have support for webpack, so it might make this easier to do neatly

hrueger · 2021-06-13T08:28:19Z

I have the same error as @chrisspiegl, however, I'm not using webpack or any other bundler. It's just plain js.

I've also made a test-repo. If you run npm start, it works but if you run npm run build and then run the dist/win-unpacked/electron-atem-connection-test.exe executable, you get the following errors in the terminal:

(node:19220) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 57)
Worker Thread error Error: Cannot find module 'A:\_Source\electron-atem-connection-test\dist\win-unpacked\resources\app.asar\node_modules\threadedclass\dist\child-process\threadedclass-worker.js'   
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:887:15)
    at Function.Module._load (internal/modules/cjs/loader.js:732:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:174:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Adding node_modules/threadedclass to the asarUnpack option of electron-builder does not help, because then the path would need to be [...]resources\app.asar.unpacked\node_modules\threadedclass[...].

Shoud I create an issue in the threadedclass repo? I think, if the user could configure the path to the threadedclass folder, it should work.

Julusian · 2021-06-13T10:33:54Z

@hrueger please try using version 3.0.0-nightly-latest-20210530-132658-16c3cdd.0. It is known that the 2.x versions do not like being packed in an asar, but the 3.0 versions are fine with it

hrueger · 2021-06-13T11:59:49Z

Thanks for the quick response, that worked 👍

lolsborn · 2021-08-26T19:47:07Z

I'm using the 3.0.0 nightly mentioned by @Julusian and am still running into issues with webpack.

(node:14441) UnhandledPromiseRejectionWarning: Error: Unable to resolve parent file path
    at eval (webpack:///./node_modules/threadedclass/dist/parent-process/threadedClass.js?:132:27)
    at new Promise (<anonymous>)
    at Object.threadedClass (webpack:///./node_modules/threadedclass/dist/parent-process/threadedClass.js?:29:12)
    at AtemSocket._createSocketProcess (webpack:///./node_modules/atem-connection/dist/lib/atemSocket.js?:81:53)
    at AtemSocket.connect (webpack:///./node_modules/atem-connection/dist/lib/atemSocket.js?:30:46)
    at Atem.connect (webpack:///./node_modules/atem-connection/dist/atem.js?:61:28)
    at eval (webpack:///./src/background.js?:24:8)
    at Module../src/background.js (/home/steven/Projects/galois/BESSPIN-Tool-Suite/besspin/cyberPhys/ui/hacker-kiosk/dist_electron/index.js:4393:1)
    at __webpack_require__ (/home/steven/Projects/galois/BESSPIN-Tool-Suite/besspin/cyberPhys/ui/hacker-kiosk/dist_electron/index.js:20:30)
    at eval (webpack:///multi_./src/background.js?:1:18)

iosimliviu · 2021-09-16T06:50:45Z

I'm also running into similar issues while running in the electron environment and trying to establish atem.connect('');.

Such as:

internal/worker.js:148 Uncaught (in promise) Error: The V8 platform used by this instance of Node does not support creating Workers
    at new Worker (internal/worker.js:148)
    at new WorkerThread (/Users/liviuiosim/Do…workerThreads.js:14)
    at Object.forkWorkerThread (/Users/liviuiosim/Do…workerThreads.js:52)
    at ThreadedClassManagerClassInternal._createFork (/Users/liviuiosim/Do…cess/manager.js:569)
    at ThreadedClassManagerClassInternal.findNextAvailableChild (/Users/liviuiosim/Do…cess/manager.js:126)
    at /Users/liviuiosim/Do…hreadedClass.js:144
    at new Promise (<anonymous>)
    at Object.threadedClass (/Users/liviuiosim/Do…threadedClass.js:29)
    at AtemSocket.<anonymous> (/Users/liviuiosim/Do…ib/atemSocket.js:90)
    at Generator.next (<anonymous>)

Have there been any updates on this topic?

sensslen · 2021-09-16T08:07:10Z

After I was able to fix the webpack stuff I also encountered this issue - just to find out that electron‘s worker thread implementation is kind of broken. At least that‘s what I read. Therefore I moved away from electron.

Julusian added the bug Something isn't working label Mar 20, 2021

sensslen closed this as completed Mar 28, 2021

Julusian reopened this May 19, 2021

hrueger mentioned this issue Jun 13, 2021

Frontend rewrite using Angular josephdadams/TallyArbiter#87

Merged

24 tasks

czenker mentioned this issue Dec 28, 2021

Problem with Atem connection wifi-tally/wifi-tally#78

Closed

Julusian mentioned this issue Jan 21, 2023

Investigate: Child thread as 'native' code #136

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

atem-connection not playing well with webpack and electron. #106

atem-connection not playing well with webpack and electron. #106

sensslen commented Mar 20, 2021

Julusian commented Mar 20, 2021

sensslen commented Mar 21, 2021 •

edited

Loading

Julusian commented Mar 21, 2021

sensslen commented Mar 21, 2021

Julusian commented Mar 21, 2021

sensslen commented Mar 21, 2021

sensslen commented Mar 23, 2021

sensslen commented Mar 23, 2021

Julusian commented Mar 23, 2021

sensslen commented Mar 28, 2021 •

edited

Loading

chrisspiegl commented May 14, 2021 •

edited

Loading

sensslen commented May 14, 2021

chrisspiegl commented May 19, 2021

Julusian commented May 19, 2021

hrueger commented Jun 13, 2021

Julusian commented Jun 13, 2021

hrueger commented Jun 13, 2021

lolsborn commented Aug 26, 2021

iosimliviu commented Sep 16, 2021

sensslen commented Sep 16, 2021

atem-connection not playing well with webpack and electron. #106

atem-connection not playing well with webpack and electron. #106

Comments

sensslen commented Mar 20, 2021

Julusian commented Mar 20, 2021

sensslen commented Mar 21, 2021 • edited Loading

Julusian commented Mar 21, 2021

sensslen commented Mar 21, 2021

Julusian commented Mar 21, 2021

sensslen commented Mar 21, 2021

sensslen commented Mar 23, 2021

sensslen commented Mar 23, 2021

Julusian commented Mar 23, 2021

sensslen commented Mar 28, 2021 • edited Loading

chrisspiegl commented May 14, 2021 • edited Loading

sensslen commented May 14, 2021

chrisspiegl commented May 19, 2021

Julusian commented May 19, 2021

hrueger commented Jun 13, 2021

Julusian commented Jun 13, 2021

hrueger commented Jun 13, 2021

lolsborn commented Aug 26, 2021

iosimliviu commented Sep 16, 2021

sensslen commented Sep 16, 2021

sensslen commented Mar 21, 2021 •

edited

Loading

sensslen commented Mar 28, 2021 •

edited

Loading

chrisspiegl commented May 14, 2021 •

edited

Loading