Gesture recognition through Mediapipe on Android

see these instructions that use Android Archive (AAR) and Gradle，The idea of gesture recognition comes from Simple Hand Gesture Recognition Code - Hand tracking - Mediapipe，Hand movement recognition

Hand Gesture

Using Mediapipe Open_CV Libraries

What this project about?

• Goal of this project is to recognize different gestures using fingers & thumb of our hand. We use the LANDMARKS output of the LandmarkLetterboxRemovalCalculator. This output is a landmark list that contains 21 landmark. Each landmark have x, y and z values. But only x, y values are sufficient for our Goal.

• We have five finger states-

• thumbIsOpen

• firstFingerIsOpen

• secondFingerIsOpen

• thirdFingerIsOpen

• fourthFingerIsOpen

• For exmaple: thumb is open if the x value of landmark 3 and the x value of landmark 4 are less than x value of landmark 2 else it is close

Installation steps

There are some following steps

• In build.gradle of App, add a Mediapipe and CameraX core library.

• In Android Manifets.xml file add permission for hand tracking in app, Camera permission, hardware Camera permission, hardware camera autofocus permission add meta data.

• Add the frame layout to display camera preview.

• Make folder of asset in app.

• 

• Make jniLibs folder.

• 

• Run the App.

Dependencies/ Labraries with their versions :

• Mediapipe- The ability to perceive the shape and motion of hands can be a vital component in improving the user experience across a variety of technological domains and platforms. We are using flogger:0.3.1, flogger-system-backend:0.3.1, findbugs:jsr305:3.0.2, guava:27.0.1-android, protobuf-java:3.11.4 versions.
• CameraX core Library- CameraX is an addition to Jetpack that makes it easier to add camera capabilities to your app. The library provides a number of compatibility fixes and workarounds to help make the developer experience consistent across many devices. We are using 1.0.0-beta10 versions.

Mainifest :

• We are using Camera permission, android.hardware.Camera permission and android.hardware.camera.autofocus permission.

What will be the output?

• We recognize gesture such as ONE, TWO, TREE, FOUR, FIVE, SIX, YEAH, ROCK, SPIDERMAN,fist and OK and We can increase/decrease volume of our device using one finger and fist.
• 

What difficulties we faced while working on project?

• Mediapipe Hands consists of two different models working together namely Palm Detection Model in which a full image is identified and it draws a box around the hand, and Hand Landmark Model operates on this boxed image formed by Palm Detector and provides high fidelity 2D hand keypoint coordinates.We have five finger states. thumbIsOpen, firstFingerIsOpen, secondFingerIsOpen, thirdFingerIsOpen, fourthFingerIsOpen. For exmaple: thumb is open if the x value of landmark 3 and the x value of landmark 4 are less than x value of landmark 2 else it is close. To detect the finger landmarks logic was quite difficult to us.

How we overcome the difficulties?

• At the beginning detecting fingers were difficult, We changes our landrmarks values and logics to detect the finger and follow this documentation from there we find ideas to detect finger using hand gesture.

What we tried to do but did not succeed?

• We tried to remove Camera preview to detect the finger but after removing preview we are not able to detect finger so we have tos how camera preview. At the end we reduce the size of camera preview.

What are our future plan regarding the development of project?

• Our future planning about this project is to control our smartphone using touchless hand gesture.