Face alignment using landmarks - Android

[ybloch]

Thank you for the amazing project!
I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code?
Are there any plans to add this?

[estebanuri]

I didn't do any face alignments in the android project. I've been testing different face alignment approaches on python scripts, and according to my tests it does improve the recognition accuracy. here is the python code that could be ported to Android (the code is inspired on the PyImageSearch site) # For dlib’s 68-point facial landmark detector: FACIAL_LANDMARKS_68_IDXS = OrderedDict([ ("mouth", (48, 68)), ("inner_mouth", (60, 68)), ("right_eyebrow", (17, 22)), ("left_eyebrow", (22, 27)), ("right_eye", (36, 42)), ("left_eye", (42, 48)), ("nose", (27, 36)), ("jaw", (0, 17)) ]) # For dlib’s 5-point facial landmark detector: FACIAL_LANDMARKS_5_IDXS = OrderedDict([ ("right_eye", (2, 3)), ("left_eye", (0, 1)), ("nose", 4) ]) def shape_to_np(shape, dtype="int"): # initialize the list of (x, y)-coordinates coords = np.zeros((shape.num_parts, 2), dtype=dtype) # loop over all facial landmarks and convert them # to a 2-tuple of (x, y)-coordinates for i in range(0, shape.num_parts): coords[i] = (shape.part(i).x, shape.part(i).y) # return the list of (x, y)-coordinates return coords def _align_face_dolap(models, image, bbox, output_size=(160, 160), eye_x=0.3, eye_y=0.35): # output_size = image.shape[1],image.shape[0] shape_predictor = models gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) x0, y0, x1, y1 = bbox rect = dlib.rectangle(left=int(x0), top=int(y0), right=int(x1), bottom=int(y1)) face_shape = shape_predictor(gray, rect) face_shape = shape_to_np(face_shape) if len(face_shape) == 68: # extract the left and right eye (x, y)-coordinates (l_start, l_end) = FACIAL_LANDMARKS_68_IDXS["left_eye"] (r_start, r_end) = FACIAL_LANDMARKS_68_IDXS["right_eye"] (n_start, n_end) = FACIAL_LANDMARKS_68_IDXS["nose"] else: (l_start, l_end) = FACIAL_LANDMARKS_5_IDXS["left_eye"] (r_start, r_end) = FACIAL_LANDMARKS_5_IDXS["right_eye"] nose_idx = FACIAL_LANDMARKS_5_IDXS["nose"] n_start, n_end = (nose_idx, nose_idx + 1) left_eye_pts = face_shape[l_start:l_end] right_eye_pts = face_shape[r_start:r_end] #nose_pts = face_shape[n_start:n_end] # compute the center of mass for each eye #left_eye_center = left_eye_pts.mean(axis=0).astype("int") #right_eye_center = right_eye_pts.mean(axis=0).astype("int") left_eye_center = left_eye_pts.mean(axis=0) right_eye_center = right_eye_pts.mean(axis=0) #nose_center = nose_pts.mean(axis=0).astype("int") # compute the angle between the eye centroids dY = right_eye_center[1] - left_eye_center[1] dX = right_eye_center[0] - left_eye_center[0] angle = np.degrees(np.arctan2(dY, dX)) - 180 #angle = 0 # compute the desired right eye x-coordinate based on the # desired x-coordinate of the left eye #left_eye_x = 0.35 # Warning!, since the face is facing front the left eye is on the right side of the image right_eye_x = eye_x left_eye_x = 1.0 - right_eye_x out_w, out_h = output_size # determine the scale of the new resulting image by taking # the ratio of the distance between eyes in the *current* # image to the ratio of distance between eyes in the # *desired* image dist = np.sqrt((dX ** 2) + (dY ** 2)) desired_dist = (left_eye_x - right_eye_x) desired_dist *= out_w #scale = 1 scale = desired_dist / dist # scale = output_size / (2 * np.linalg.norm(nose_center)) # compute center (x, y)-coordinates (i.e., the median point) # between the two eyes in the input image eyes_center = ((left_eye_center[0] + right_eye_center[0]) / 2, (left_eye_center[1] + right_eye_center[1]) / 2) # # grab the rotation matrix for rotating and scaling the face # H = cv2.getRotationMatrix2D(eyes_center, angle, scale) # # # update the translation component of the matrix tX = out_w * 0.5 #tY = out_h * eyes_center[1] tY = out_h * eye_y # H[0, 2] += (tX - eyes_center[0]) # H[1, 2] += (tY - eyes_center[1]) H[0, 2] += tX - eyes_center[0] H[1, 2] += tY - eyes_center[1] ret = cv2.warpAffine(image, H, output_size) # return the aligned face return ret

…

On Mon, Sep 21, 2020 at 2:17 AM ybloch ***@***.***> wrote: Thank you for the amazing project! I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code? Are there any plans to add this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFVH2LZHBJ5XMYK7AB6LYTSG3OVBANCNFSM4RUDB2UQ> .

[ybloch]

I think the reason you did not see a significant improvement is because you did not use the same method they used in the original insight-face repo ...
From tests I did in Python on the original repo, I remember it had a significant effect on accuracy.

I saw in this repo he made alignment in their same method, with mtcnn, and their 5 landmarks is:
right eye, left eye, nose, left side mouth, right side mouth.

Maybe if you try to use his method in your project you will see significant improvement...

[estebanuri]

Hi ybloch, as I said, the face alignment *does* help to gain some accuracy. I looked at the insight face alignment code a few months ago, and there were several methods implemented. I tried some, and they did not work properly. Following your comment, I looked at the repository again and found this alignment code <https://github.com/deepinsight/insightface/blob/master/common/face_align.py>, that is different from what I had originally seen. If I'm not mistaken the code warps the image to match the landmarks with established positions. I did several tests and found better results with the code that I shared in the previous email, that is similar to the one that is used in the repo you shared with me <https://github.com/syaringan357/Android-MobileFaceNet-MTCNN-FaceAntiSpoofing/blob/master/app/src/main/java/com/zwp/mobilefacenet/mtcnn/Align.java>, which simply rotates the image to align the eyes and then crops the centered image. But perhaps, I'm doing something wrong while applying the insight-face method. <https://github.com/syaringan357/Android-MobileFaceNet-MTCNN-FaceAntiSpoofing/blob/master/app/src/main/java/com/zwp/mobilefacenet/mtcnn/Align.java>

…

On Mon, Sep 21, 2020 at 12:38 PM ybloch ***@***.***> wrote: I think the reason you did not see a significant improvement is because you did not use the same method they used in the original insight-face repo <https://github.com/deepinsight/insightface> ... From tests I did in Python on the original repo, I remember it had a significant effect on accuracy. I saw in this repo <https://github.com/syaringan357/Android-MobileFaceNet-MTCNN-FaceAntiSpoofing> he made alignment in their same method, with mtcnn, and their 5 landmarks is: right eye, left eye, nose, left side mouth, right side mouth. Maybe if you try to use his method in your project you will see significant improvement... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFVH2N3UU4VAZETL5LG47TSG5XNZANCNFSM4RUDB2UQ> .

[ybloch]

Hello Uri,
Sorry, I did not read your first comment carefully... ;)
So we conclude that it has a positive effect, do you have any plans to add face alignment to your Android project?

[raja259]

Hi

It's a request to add face alignment to your Android project as this will increase the accuracy and also while taking picture for the recognition we can check that both eyes both ears forehead and chin landmark should have good values so recognition will take less time and more accurate and picture captured is good enough for the recognition purpose.

Currently I am facing lot of problems as side photos are getting captured and those are not recognizable and there matching criteria is higher then .75 which we have made default.

Thanks in advance

[ybloch]

Hello @estebanuri,
can you please share with me the python script you used for testing the MobileFAceNet TF-lite model?