feat: config plugin for android gpu libraries

[TkTioNG]

Allow setting needed gpu library for expo/expo-managed app through expo config plugins.

NOTE

• Not sure about the api that allow multiple libraries as an array of string, it just feels bad
• I just test on my pixel and samsung devices only, a quick pr being drafted
• I don't even sure if uses-native-library even works on expo-managed app, i think it don't? Unless it works, else it is a WIP.

fix: #84 (comment)

[mrousavy]

this is huge!! thanks so much - lmk when it's ready :)

[lucksp]

as per ESLint rule eqeqeq:

It is considered good practice to use the type-safe equality operators === and !== instead of their regular counterparts == and !=.

[lucksp]

I don't even sure if uses-native-library even works on expo-managed app, i think it don't? Unless it works, else it is a WIP.

Yes, I am not 100% sure either as noted in my comment #84 . How did you test this, on your android devices - bare or expo? Did you see any performance difference?

[lucksp]

Thanks for starting this...I have tried my best to test this inside Expo:

• I copied your changes to my local node_modules files
• updated app.config to reference plugin
• ran patch-package to bring in the changes, and ran app via npx expo run:android --device

My model will not load at all if I include any delegate options to loadTensorflowModel()

I have also tried different options passed into the app.config#plugin for this lib.

[TkTioNG]

Thanks for starting this...I have tried my best to test this inside Expo:

* I copied your changes to my local `node_modules` files

* updated app.config to reference plugin

* ran `patch-package` to bring in the changes, and ran app via `npx expo run:android --device`

My model will not load at all if I include any delegate options to loadTensorflowModel()

I have also tried different options passed into the app.config#plugin for this lib.

Hey, thanks for trying it. I guess if just copy the change to node_modules, only the .ts files get updated, but the expo config is read from exported .js file.

I personally test it with:

1. Generate the package changes with yarn prepack
2. yarn link in the local react-native-fast-tfllite repo
3. yarn link react-native-fast-tflite in the testing app repo
4. Rebuild/ran the app without cache

Not sure, if that helps. I think I might have time later today and have a better look

[lucksp]

I personally test it with:

1. Generate the package changes with yarn prepack
2. yarn link in the local react-native-fast-tfllite repo
3. yarn link react-native-fast-tflite in the testing app repo
4. Rebuild/ran the app without cache

Not sure, if that helps. I think I might have time later today and have a better look

I am unable to find your branch... @TkTioNG did you make a commit from a fork?

[TkTioNG]

I personally test it with:

1. Generate the package changes with yarn prepack
2. yarn link in the local react-native-fast-tfllite repo
3. yarn link react-native-fast-tflite in the testing app repo
4. Rebuild/ran the app without cache

Not sure, if that helps. I think I might have time later today and have a better look

I am unable to find your branch... @TkTioNG did you make a commit from a fork?

Yes, it is from a fork, I tested it with expo app and it works.

[lucksp]

ok, yes! I copied the repo and added your changes, then link'ed to my project. the AndroidManifest.xml file shows:
<uses-native-library android:name="libOpenCL.so" android:required="false"/> when enableAndroidGpuLibraries: true, in app.config! 🎉
making progress :)

Now that it's linking, I have tried to load the model with GPUDelegate android-gpu or nnapi:

const model = await loadTensorflowModel(
      {
        url,
      },
      'android-gpu' // or 'nnapi'
    );

however, the model never loads if I pass an android-gpu...it gets stuck. Do you see this too?

I am on "react-native-fast-tflite": "~1.3.0" - anything newer fails to build due to #91 and #99

I am using a real 💩 Android Blu G53 device so it struggles with anything, even browsing web

Notes related to app.config settings:

• enableAndroidGpuLibraries: true:

  • nnapi will load the model
    • no perceived performance improvement.
  • android-gpu will not load the model. Do you see this too?

• enableAndroidGpuLibraries: ['libOpenCL-pixel.so', 'libGLES_mali.so'],:

  • nnapi will load the model
    • app freezes, I can hear the camera turning on & off with audible clicks. This seems to make it worse 🤷🏻‍♂️
  • android-gpu will not load the model. Do you see this too?

I am surprised that performance wasn't improved with the default enableAndroidGpuLibraries: true flag enabled. Maybe it's still an improvement somewhere?

How do I know if I need GPU enabled settings?

I don't have time to test out a preview build if performance is improved outside of development environment...I also don't think I can build with a linked package...

[TkTioNG]

@lucksp
OK, I think the config plugin does work.

I believe your case is more of a GPU vendor support. I just did a check, that BLU G53 is using PowerVR GPU, and the vendor OpenCL support is libPVROCL.so or libpocl.so

So may be you can try with enableAndroidGpuLibraries: ['libPVROCL.so'], if it is still not working may be you can try edit the manifest manually to only include the libPVROCL.so but not libOpenCL.so.

When trying, you can try to look at the android log, it at least can tell you something.

And regarding the nnapi do not has performance improvement, it might be most operations is falling back to use CPU because the nnapi module not supporting it, it just a guess, but you can also check it out in the android log.

There's a way to check for the vendor provided lib name too, by using device explorer in android studio. But there's not standard directory it store, it could be inside system/lib, system/vendor/lib, lib64, or something like that.

[lucksp]

Thanks for the info.

Since there’s so many different options of Android devices, is there a way to know how to best use the config setting with our negatively impacting some devices by accident?

any thoughts on the why android-gpu will not load the model? Do you have this issue?

[TkTioNG]

Well, I can't answer you why the android-gpu don't work though, gotta look at the log for better idea, for my previous case, where I try to add android GPU support, my devices told me that, the default gpu library (OpenGL), does not support all operations of tflite, so it failed. Since, tflite are using OpenCL, so I decided to add the OpenCL vative library to the app, so the device can handle it.

I was unaware of including other GPU library, may affect the entire loading though, this needs more checking, may be because when you include libGLES_mali.so, the device load using that lib just to find out that it don't work as it is vendor-specified, i don't know need to look the the log of before and after to compare it.

I think that better case will be include a working library directly inside the package, and load it, or may be scan the device vendor support to select the proper lib. But it is much more complicated

[lucksp]

Thank you. All makes sense.

[lucksp]

Does as intended. Device support varies which is separate issue.

Looks good to me @mrousavy

[lucksp]

I do think there’s concern to release this because any release after 1.3.0 fails to build due to: #91 and #99

EDIT: I think #112 should resolve

[lucksp]

jsut to update, that this PR + #112 is working well when I set my GPUDelegate to nnapi. Likely device dependent, sure, but the performance is so noticeably improved on my crappy G53.

I am going to patchPackage this change on my app for the time being to see how it does with some testers.

Thank you @TkTioNG