Gesture Recognition is an important part of Human Computer Interaction (HCI).

This article was originally published on <a href="https://becominghuman.ai/a-technical-introduction-to-gesture-recognition-for-data-scientists-9ea4f1b76ed4">becominghuman.ai</a> on the 27 February 2020.This is part of a project at NordAxon, the company where I work at as a Data Scientist. The purpose of this article is to share my insights and knowledge, meaning to instill collaboration within the Data Science community and also to help you kickstart your own project in gesture recognition. <h2>Background </h2>Gesture Recognition is an important part of Human Computer Interaction (HCI). It can be used to improve user interfaces, e.g. virtual reality in the gaming industry or remote control of robot arms using gestures. It can even make life easier for people with disabilities as an assistive technology such as sign language translation.<img class="image_resized" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fwalkinghuman.png?alt=media&amp;token=55309706-55f1-43c5-8165-202779bb5d99" />Example problem formulation: Classify the type of walk a person is doingAs of today, with Deep Learning and only 2D RGB images as inputs, we can build fairly robust models for static gesture recognition. Though, the more promising application for continuous gestures recognition still faces many challenges and is an open problem.  <h2>Prerequisite </h2>This article is intended for Data Scientists who have at least basic understanding of machine learning and deep learning in computer vision. <h4>Challenges to overcome </h4>Most problems in gesture recognition are associated with accuracy and performance. For instance, different people might perform the same gesture differently in both speed and movement range. This greatly increases the complexity of the problem if we have many types of gestures. In this article, we will restrict our discussion only to hand gestures as they are the <a href="http://personal.ee.surrey.ac.uk/Personal/S.Hadfield/papers/Camgoz17.pdf">most important features</a> for many gesture recognition tasks. <ul><li>Occlusions (i.e. hands not in full sight) occur frequently and might also be a challenge to model. A robust model must be able to take into account the global- and local features in order to handle occlusions. Imagine a person that is holding a banana, then the part of the hand behind the banana is not visible to the camera/model.</li></ul><img class="image_resized" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fbananaman.jfif?alt=media&amp;token=cbf20131-4772-461a-81ea-87331de72e42" />Illustration of keypoint detection of hands.<ul><li>Co-articulation occurs when a gesture is performed differently depending on what gesture comes before and after. Co-articulation usually happens between two gestures as an overlap. This occurs frequently in fluent sign language. </li><li>A limiting aspect for creating an end-to-end software is the trade-off between real-time performance and accuracy. This may be a requirement for creating e.g. reliable gesture user interfaces for controlling robot arms. </li><li>Another problem with continuous gesture recognition is that it is expensive to create datasets with frame-level annotations, which may be necessary for sign language recognition or other fast-paced gesticulations. Imagine any sign language with over 1000 words (understatement) and the amount of time that must be spent annotating this dataset. </li></ul> <h2>Creating your own datasets </h2>Let's address the problem of acquiring a frame-level dataset for continuous hand gesture recognition. This problem was addressed by a <a href="https://openaccess.thecvf.com/content_cvpr_2016/papers/Koller_Deep_Hand_How_CVPR_2016_paper.pdf">paper from 2016</a> where the authors train a CNN on 1 000 000 images using weakly supervised learning. Their algorithm takes advantage of the Expectation-Maximization (EM)-algorithm and "inaccurate" labels in order to train a fairly robust CNN model on different hand shapes. In other words, through clustering the model learns different hand shapes. This is a times-saving approach if you want to create your own frame-level dataset for handshapes. <img src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Farrows.png?alt=media&amp;token=e8999879-98ea-4a36-b495-b2b8c174d3f7" />Illustration of weak annotations/labels of the frames in a video. Each frame is an image. <h2>Deep Learning Architectures </h2>The gestures are characterized by spatial movements of the hand through the time, therefore our model must be able to capture spatio-temporal features. In other words, the model will handle 2D images (spatial features) through time (temporal features). Here I list some approaches that can be explored and tested: <ul><li>3D Convolutional Neural Networks (3D CNN), the author of this <a href="https://arxiv.org/abs/1412.0767">paper</a> argues that this type of model is more suitable for modelling spatio-temporal features compared to 2D CNNs. These models perform well when doing action recognition on videos and their usage may be extended to continuous gesture recognition. For a demonstration of these type of networks, skip to 18:00 in the following presentation.</li></ul><a href="https://www.youtube.com/watch?v=keffWSqi67w">https://www.youtube.com/watch?v=keffWSqi67w</a>The presentation was made by TwentyBN for a PyData conference. Skip to 18:00 in the video to see the demonstration of their model which is partly based on 3D convolutional layers.<ul><li>Object localization models which predicts bounding boxes such as YOLOv3, Single Shot Detectors (SSD) or Faster R-CNN can be used for static hand gestures recognition (small demonstration of YOLOv3 in the figure below). You can choose between fast real-time inference models e.g. YOLOv3 and SSD. Else if you value accuracy over speed you can use R-CNNs.</li></ul><img class="image_resized" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fdavid.jfif?alt=media&amp;token=460c8c96-3669-4cc4-972f-ab5f9c00b42e" /><ul><li>Keypoint detection models  can estimate joint positions which in turn model human poses. These systems are usually built on top of an object localization model and their output could be used as features for lightweight classification models. For instance, with the help of Hidden Markov Models (HMM) or Recurrent Neural Networks (RNN). Alternatively, there exists complete end-to-end systems such as MediaPipe, OpenPose and wrnchAI. Check them out! You might be able to integrate these solutions into your gesture recognition system if you don't want to develop one yourself.</li></ul> <h2>Summary </h2><ul><li>In this article I have introduced you to some of the challenges in continuous hand gesture recognition. For instance, occlusions and co-articulations. </li><li>You can use weakly supervised learning to train your model and therefore save time. </li><li>An end-to-end system could be a combination of different systems such as OpenPose + RNN (example) for continuous gesture recognition. </li></ul> <h2>Next steps </h2>The next steps are to decide the following <ul><li>What will your dataset look like (which hand gestures) and how will you gather it? </li><li>Compare all the different approaches/models. </li></ul>