Gesture Recognition is an important part of Human Computer Interaction (HCI).

This article was originally published on <a href="https://becominghuman.ai/a-technical-introduction-to-gesture-recognition-for-data-scientists-9ea4f1b76ed4">becominghuman.ai</a> on the 27 February 2020.This is part of a project at NordAxon, the company where I work at as a Data Scientist. The purpose of this article is to share my insights and knowledge, meaning to instill collaboration within the Data Science community and also to help you kickstart your own project in gesture recognition.&nbsp;<h2>Background&nbsp;</h2>Gesture Recognition is an important part of Human Computer Interaction (HCI). It can be used to improve user interfaces, e.g. virtual reality in the gaming industry or remote control of robot arms using gestures. It can even make life easier for people with disabilities as an assistive technology such as sign language translation.<img class="image_resized" style="width:46.7%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fwalkinghuman.png?alt=media&amp;token=55309706-55f1-43c5-8165-202779bb5d99">Example problem formulation: Classify the type of walk a person is doingAs of today, with Deep Learning and only 2D RGB images as inputs, we can build fairly robust models for static gesture recognition. Though, the more promising application for continuous gestures recognition still faces many challenges and is an open problem.&nbsp;&nbsp;<h2>Prerequisite&nbsp;</h2>This article is intended for Data Scientists who have at least basic understanding of machine learning and deep learning in computer vision.&nbsp;<h4>Challenges to overcome&nbsp;</h4>Most problems in gesture recognition are associated with accuracy and performance. For instance, different people might perform the same gesture differently in both speed and movement range. This greatly increases the complexity of the problem if we have many types of gestures. In this article, we will restrict our discussion only to hand gestures as they are the <a href="http://personal.ee.surrey.ac.uk/Personal/S.Hadfield/papers/Camgoz17.pdf">most important features</a> for many gesture recognition tasks.&nbsp;<ul><li>Occlusions (i.e. hands not in full sight) occur frequently and might also be a challenge to model. A robust model must be able to take into account the global- and local features in order to handle occlusions. Imagine a person that is holding a banana, then the part of the hand behind the banana is not visible to the camera/model.</li></ul><img class="image_resized" style="width:52.58%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fbananaman.jfif?alt=media&amp;token=cbf20131-4772-461a-81ea-87331de72e42">Illustration of keypoint detection of hands.<ul><li>Co-articulation occurs when a gesture is performed differently depending on what gesture comes before and after. Co-articulation usually happens between two gestures as an overlap. This occurs frequently in fluent sign language.&nbsp;</li><li>A limiting aspect for creating an end-to-end software is the trade-off between real-time performance and accuracy. This may be a requirement for creating e.g. reliable gesture user interfaces for controlling robot arms.&nbsp;</li><li>Another problem with continuous gesture recognition is that it is expensive to create datasets with frame-level annotations, which may be necessary for sign language recognition or other fast-paced gesticulations. Imagine any sign language with over 1000 words (understatement) and the amount of time that must be spent annotating this dataset.&nbsp;</li></ul>&nbsp;<h2>Creating your own datasets&nbsp;</h2>Let's address the problem of acquiring a frame-level dataset for continuous hand gesture recognition. This problem was addressed by a <a href="https://openaccess.thecvf.com/content_cvpr_2016/papers/Koller_Deep_Hand_How_CVPR_2016_paper.pdf">paper from 2016</a> where the authors train a CNN on 1 000 000 images using weakly supervised learning. Their algorithm takes advantage of the Expectation-Maximization (EM)-algorithm and "inaccurate" labels in order to train a fairly robust CNN model on different hand shapes. In other words, through clustering the model learns different hand shapes. This is a times-saving approach if you want to create your own frame-level dataset for handshapes.&nbsp;<img src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Farrows.png?alt=media&amp;token=e8999879-98ea-4a36-b495-b2b8c174d3f7">Illustration of weak annotations/labels of the frames in a video. Each frame is an image.&nbsp;<h2>Deep Learning Architectures&nbsp;</h2>The gestures are characterized by spatial movements of the hand through the time, therefore our model must be able to capture spatio-temporal features. In other words, the model will handle 2D images (spatial features) through time (temporal features). Here I list some approaches that can be explored and tested:&nbsp;<ul><li>3D Convolutional Neural Networks (3D CNN), the author of this <a href="https://arxiv.org/abs/1412.0767">paper</a> argues that this type of model is more suitable for modelling spatio-temporal features compared to 2D CNNs. These models perform well when doing action recognition on videos and their usage may be extended to continuous gesture recognition. For a demonstration of these type of networks, skip to 18:00 in the following presentation.</li></ul><a href="https://www.youtube.com/watch?v=keffWSqi67w">https://www.youtube.com/watch?v=keffWSqi67w</a>The presentation was made by TwentyBN for a PyData conference.&nbsp;Skip to 18:00 in the video&nbsp;to see the demonstration of their model which is partly based on 3D convolutional layers.<ul><li>Object localization models which predicts bounding boxes such as YOLOv3, Single Shot Detectors (SSD) or Faster R-CNN can be used for static hand gestures recognition (small demonstration of YOLOv3 in the figure below). You can choose between fast real-time inference models e.g. YOLOv3 and SSD. Else if you value accuracy over speed you can use R-CNNs.</li></ul><img class="image_resized" style="width:42.9%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FGesture%20recognition%2Fdavid.jfif?alt=media&amp;token=460c8c96-3669-4cc4-972f-ab5f9c00b42e"><ul><li>Keypoint detection models  can estimate joint positions which in turn model human poses. These systems are usually built on top of an object localization model and their output could be used as features for lightweight classification models. For instance, with the help of Hidden Markov Models (HMM) or Recurrent Neural Networks (RNN). Alternatively, there exists complete end-to-end systems such as MediaPipe, OpenPose and wrnchAI. Check them out! You might be able to integrate these solutions into your gesture recognition system if you don't want to develop one yourself.</li></ul>&nbsp;<h2>Summary&nbsp;</h2><ul><li>In this article I have introduced you to some of the challenges in continuous hand gesture recognition. For instance, occlusions and co-articulations.&nbsp;</li><li>You can use weakly supervised learning to train your model and therefore save time.&nbsp;</li><li>An end-to-end system could be a combination of different systems such as OpenPose + RNN (example) for continuous gesture recognition.&nbsp;</li></ul>&nbsp;<h2>Next steps&nbsp;</h2>The next steps are to decide the following&nbsp;<ul><li>What will your dataset look like (which hand gestures) and how will you gather it?&nbsp;</li><li>Compare all the different approaches/models.&nbsp;</li></ul>

A Technical Introduction to Gesture Recognition for Data Scientists

Current status on NordAxon's solution Emely, a tool for practicing speaking interactively

This article was originally <a href="https://medium.com/@nordaxon/emely-speaking-practice-with-focus-on-interaction-3c79ec2a5468">published on medium</a> on the 14 September 2022.<h2>Introduction</h2>There are many tools available for learning new languages, where the focus usually lies on learning words, grammar, and pronunciation. During 2020, Helsingborg AMF together with&nbsp;<a href="https://www.nordaxon.com/">NordAxon</a> identified additional barriers to learning a new language related to interaction. One of the best ways to learn a new language is to speak it and interact with others that are also trying to learn or that are already fluent speakers. This can be very challenging for people that have limited access to conversation partners, limited time to seek these out, or simply do not have the courage to do so.Having a job is an efficient way to learn a local language since it often involves a lot of interaction, but to get a job it is usually required that you are already a fluent speaker. The idea brought together by Helsingborg AMF and NordAxon is to make interactive conversation practice more accessible, in the same way as other parts of language practice have been made accessible already.<img class="image_resized" style="width:100%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely-Interactive%20Speaking%20Practice%20using%20ML-powered%20Conversational%20Agent%2Femely-screen.jpg?alt=media&amp;token=6473b5ab-1a19-47f2-8a7d-3d9cf43ee355"><h2>&nbsp;</h2><h2>Emely</h2>Emely is a tool for conversation practice made available as a web application. The first iterations of Emely have been focused on job interview practice and common conversations on varying levels of difficulty. The user interacts with Emely through a chat-like interface, where the user records their response, and Emely in turn responds through speech synthesis via the microphone, as well as displaying the response in the chat as text.Since Swedish is not yet a “high resource” language within the field of NLP (Natural Language Processing), we are for now using English as an intermediary language for certain parts of the functionality (“high resource” in this context means that there would be many useful datasets and previous modelling work performed in Swedish). In the image below we show the architecture of the conversation flow that is used in the current version of Emely. A response is generated through the steps 1-6 indicated in the image. The steps are:<ol><li>Speech recognition – turns the user input into text.</li><li>Translation – we make use of translation from Swedish to English.</li><li>Dialog control – we control the flow of dialogs differently based on scenarios, and how the conversation progresses.</li><li>Generative Dialog Model – a machine learning model generates an answer using user input, previous conversation history, as well as additional information from the dialog control.</li><li>Translation&nbsp;– the response from Emely is translated back into Swedish.</li><li>Speech synthesis – text is transformed to speech which is then played on the user’s device.</li></ol><img class="image_resized" style="width:76.49%;" src="https://ik.imagekit.io/nordaxon040/tr:w-1000/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely%20%E2%80%93Speaking%20Practice%20with%20focus%20on%20Interaction%2Farchitecture.png?alt=media&amp;token=189af527-f4e6-47dc-9c8b-16bae6e18675">In addition to these steps, we analyze conversations, considering both how well a user is performing, and how well Emely is performing. The analysis of user replies can be visualized in the user interface, and further analyzed for improving language skills by yourself, in collaboration with a language coach/teacher, or in the case of job interviews a job coach.<img class="image_resized" style="width:100%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely-Interactive%20Speaking%20Practice%20using%20ML-powered%20Conversational%20Agent%2Femely-screen2.jpg?alt=media&amp;token=26f6ce9c-8c64-4f5c-92c5-b95c41d77d4b"><h2>&nbsp;</h2><h2>Job Interviews</h2>Large amounts of resources in Swedish municipalities are spent on helping people being able to find a job. Language skills is a large part of this task, where job coaches are working together with language instructors and teachers to help people improve their interactive language skills.Having job interviews is very different to having common conversations. One big difference is that the conversations are usually only being controlled by the interviewer, who is asking many questions and follow-up questions. To make a good impression at a job interview it is in most cases not enough to only respond to questions that are easily expected, but there might be an even higher weight on how you can answer on follow-up questions. Also, an interview can switch topics fast, but are usually centered around the person being interviewed.The Interview module in Emely was developed in collaboration with job coaches at Helsingborg AMF, ensuring that the above aspects were taken into consideration. The main idea in the current version of Emely is to steer interviews in the “dialog control” and utilize a custom generative dialog model that is specialized in asking follow-up questions. Therefore, the model has been trained using a large dataset with examples of good follow-up questions. An example of user responses and “labels” (a label is an example of what we want the model to respond in this specific context) from the training dataset is shown below:Emely: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;How do you think a good boss should be?User response: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A good manager is clear and coaching, gives responsibility and lets his employees come up with ideas and views.Emely: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;How should a boss not be?User response: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A manager should not be afraid of conflict or inaccessible.Emely: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;What can happen then?&nbsp;<h2>Research</h2>Several of the components in Emely is currently subject to research in the field of NLP. NordAxon is both actively engaged in this research relating to some of these components, as well as following the state of the art in NLP closely.One topic where NordAxon has been involved in research is in the quality assurance of generative dialog models. Quality assurance in this case is mainly focused on analyzing the quality of the generated responses and conversations. In other words, how can we determine that Emely is responding reasonably within the context, grammatically correct, as well as having overall good conversations.An initial framework for evaluating conversations was developed in a collaboration between NordAxon, Research Institutes of Sweden (RISE), and Lund University. A research article from this collaboration can be found <a href="https://arxiv.org/abs/2203.15414">here</a>, and an initial implementation has been made available as open source <a href="https://github.com/NordAxon/GDM-testing">here</a>.&nbsp;<h2>Emely on H22</h2>In the summer of 2022, Helsingborg held an innovation exhibition named H22 City Expo. The exhibition is described as “An international event that focused on future welfare and urban development solutions and how we can create a smarter, more sustainable, and thoughtful city – together” [<a href="https://www.h22cityexpo.se/">https://www.h22cityexpo.se/</a>]. A main attraction on the exhibition was speaking with Emely.During H22 City Expo more than 600 visitors initiated over 2.500 conversations with Emely. This meant that visitors spent in total 86 hours talking with Emely, sending 10.742 replies. Even the Swedish crown princess <a href="https://www.mynewsdesk.com/se/helsingborg/pressreleases/3-smarta-roliga-och-fiffiga-loesningar-3186558">Victoria had a quick chat with Emely</a>.The H22 City Expo was therefore an excellent opportunity for us to evaluate the current version of Emely and analyze user interactions. For example, it has helped to improve performance, UI/UX, and handling of the very varying quality of messages that is received at an exhibition.&nbsp;<h2>Prospects</h2>Moving forward, we are looking to expand the scope and usage of Emely in the following ways:<ul><li>User group – who could benefit from interactive language practice?</li><li>Practice scenario – which scenarios are relevant for practicing interactively?</li><li>Language – which languages other than Swedish could be useful to use with Emely?</li></ul>If you are interested in the functionality of Emely, or how Emely can be applied to practice speaking interactively in other scenarios, please get in touch with us at info.nordaxon.com.

Emely: GPT-driven service for conversational training

Analysis, modelling, and challenges with biometric data. Use case: Nystagmus, a medical condition causing involuntary eye movements.

<h2>Introduction</h2>Just like with music, clothing, and food trends there are also trends in our world of AI (Artificial Intelligence) and ML (Machine Learning). Predictive analysis of biometric data has increased in popularity over the last years, a clear trend in the AI/ML community. This is for example the data which is used by casual runners to measure their performance or by forensic scientist to investigate crime. Nurses and medical doctors use biometric data to evaluate the condition of patients, and it can of course even be used to authenticate the identify of a person with a high level of accuracy. In this article, I will take a look at analysis of biometric data, the approaches that are used to model the data, challenges that data scientist may encounter when dealing with this type of data and relate all those to the use case Nystagmus, a medical condition causing involuntary eye movements.&nbsp;<h2>What is biometric data?</h2>Before we can conduct an in-depth analysis of biometric data, we first need to know what they are. The word biometric consists of two components:&nbsp;bio&nbsp;meaning life, and&nbsp;metric,&nbsp;relating to a standard of measurement. Basically, biometric data is any information that can be measured from a human body. When we think of biometric data we might think of respiration rate, heart activity and brain activity, but biometric data may also include a person’s voice or their vein structure.&nbsp;<h2>Biometric data vs personal data</h2>Some of you might ask yourself, what is the difference between biometric data and personal data? As discussed above, biometric data are data that can be measured from a person. Personal data on the other hand, are data that can be used to determine the identity of a person. Examples of personal data include the name of person, a social security number, a home address, a telephone number, and medical records&nbsp;[1]. For example, the fingerprint of a person is biometric data, but it is also personal data, since it can be used to identify a person. This means that biometric data is in many cases also personal data. When dealing with biometric data, it is therefore important to treat it with care, and I will come back to how to handle this later in the article.&nbsp;<h2>Machine learning</h2>Machine learning is, as we all know, one of the hottest topics in the world of research, science, computers, and just about everywhere data is talked about. This technology can provide remarkable insight to the most complex of problems, and it is therefore an incredible tool for us data enthusiasts. When working with biometric data, it is useful since biometric data in many cases is quite complex. Here is a summary of the two most commonly types of machine learning approaches used today:&nbsp;supervised learning&nbsp;and&nbsp;unsupervised learning.Supervised learning is used to establish a relationship between some input, and an output, e.g., the max pulse and the age of a person. When building supervised learning models, input data is collected with corresponding outputs, and the model is evaluated based on how well it can describe this relationship.Unsupervised models, on the other hand, are used on data where there is no corresponding output. There are two applications for unsupervised learning that are frequently used:&nbsp;dimensionality reduction&nbsp;and&nbsp;clustering. The aim of dimensionality reduction is to extract the relevant data in datasets where the input data have many dimensions. Dimensionality reduction is particularly useful if the number of input dimensions is large.Clustering is used to find data that have similar properties. Unsupervised learning can be used as an analysis method all by itself, or as a pre-processing step to improve the performance of supervised learning models. We can think of unsupervised learning as a way of sorting our data.Deep learning is a modelling approach used to create complex models that can find patterns in data that are otherwise hard to find, and it can be used for both supervised and unsupervised machine learning problems.&nbsp;<h2>A case with eye movement signals</h2>I will now go through an example of how to use machine learning on biometric data from my research as a PhD student. I studied a condition called&nbsp;nystagmus, and patients who suffer from this express involuntary eye movements. There are various reasons why a person might have nystagmus, and in many cases, it is a bit on an enigma for medical doctors working with the patients. The goal of my research was to give these medical professionals a way to accurately evaluate the condition of their patients, and to measure and predict the development of the patients’ eye movements. Below you can see an example of a person expressing horizontal nystagmus [2].<img class="image_resized" style="width:31.26%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBiometric%2Feye.gif?alt=media&amp;token=34f8a5c9-2f02-4829-841e-b500f27ef83e">In order to build our models, we need some data, and the first thing I did, was to record eye movements of nystagmus patients. This was done with a technique called&nbsp;eye tracking, which films the eyes of the person, and with the help of image analysis, estimates where the person looked. The outputs from the eye tracker are time dependent horizontal and vertical eye positions. The next step was to model these eye movements, and this was done using a frequency domain analysis called Fourier transforms. This is a form of supervised learning, where the goal is to estimate the original signal using a summation of sinusoidal components. From the Fourier analysis, I was able to extract characteristics, or&nbsp;features&nbsp;as they are called in the machine learning world, that describe the eye movements for each of the participants in my studies.For some nystagmus patients, the eye movements change based on the how the eyes are angled. For example, the eye movements were different when the person looked to the right compared to when the person looked to the left. Therefore, I wanted to compare the eye movements from one recording to another recording. To do so, I used an unsupervised learning clustering method to compare the features that I got from the Fourier analysis. The result was a method that could distinguish between eye movements from a person, without having to go through the eye movement recordings manually. The method could also be used to compare eye movements from different people. Now, lets imagine that we have a dataset of 1000 people, recorded at different locations by different people. By running my algorithm, we could find the people that have similar eye movement patterns. This is an amazing result, as it lets us compare data from different patients, recorded in different years, at different locations — without having to consult an ophthalmologist!Once the algorithm was developed, we could use it to link results to things like treatments or patient diagnostics. This could for example be used to investigate which treatments that gave the best effect to a certain type of eye movement pattern. This type of analysis could just as well have been applied to heart activity data (ECG), or just about any other form of biometric time series data, and it shows how powerful machine learning can be.&nbsp;<h2>Machine learning on biometric data - challenges and solutions</h2>As we have seen, machine learning is quite useful for analyzing biometric data, and the applications using machine learning will likely have an exponential increase in the years to come. However, there are many challenges when working with biometric data, as most often these data are treated as personal data. In some cases, it is possible to anonymize the data, so that it is no longer possible to link the data to a specific person. But it is not always guaranteed that the data will no longer be classified as personal data.Within the European Union there is the General Data Protection Regulation (GDPR), which regulates how data may be stored and who will have access to the data. Biometric data are of course included here, and this will sometimes put limitations on how models can be developed and how data are allowed to be shared. In addition to GDPR, the European Parliament released a document on the 21:st of April 2021 titled:&nbsp;Laying down harmonized rules on artificial intelligence and amending certain union legislative act. This document is focused on ensuring that AI systems are safe and trustworthy. In the coming years, this will be a hot button topic, and it will be important for data scientist and others interested in machine learning working with biometric data.To run experiments with personal data, it is necessary to have consent from each participant, even if the data has been anonymized, and in the case of research, an ethical approval is in most cases required. This can lead to practical problems when using supervised learning because it requires annotated data. To get these annotations, an expert must go through the data and provide the invaluable annotations. When developing machine learning models, this is a problem, since most models need a lot of data to give good results. This is especially true when working with complex models like deep neural networks. If data cannot be shared, then a lot of hours needs to be devoted to data annotation for each project, and it is difficult to compare performance between different methods. If the data collection process is slowed down, it may hurt the development of this useful technology. In some cases, there are open datasets where the participants of research studies have approved that their data are shared. One example of an open-source database for biometric data is Physionet [3], which consists of a wide variety of medical recordings.&nbsp;<h2>Creating our own data</h2>What do we do if we don’t have enough data to build or models? If we can’t collect more data, or if we need our data now, there are typically two approaches that we can use. The first alternative is to create&nbsp;synthetic data. A synthetic dataset is just what it sounds like, data that have been synthesized. Here it is possible to either use a model that can generate the type of data that are desired, or to manually create data that resembles the original dataset. Let’s say that that we are interested in analyzing a population of people, where we know their age and max pulse. There are recordings from 100 people, but we believe that we would need twice as much for our model development. By estimating the statistical distribution of the 2-dimensional data (age and max pulse), we could sample from the estimated distribution, and thus create “new” data points. The distribution could for example be estimated using a Kernel density estimation (KDE) method. This approach could also be used to create “fake” medial journals with realistic values. The benefit of this is that we don’t need to give away any real medical data to someone who is in need of testing an application on medical journal data.The second option is to use&nbsp;data augmentation, which is a method for creating “new” data from a dataset that already exists. This is a common method for image data, where the images can be rotated, flipped, or noise can be added. Let’s say that we are working on a method for detecting early-stage brain tumors, and we have used computer tomography (CT) to scan a number of patients and healthy controls. We have some data, but as always when developing machine learning models, we would like to have more, and we decide to use data augmentation to create more images. In the example below, a brain image from a CT scan [4]&nbsp;has been augmented using three different methods.<img src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBiometric%2Fskull.png?alt=media&amp;token=b41af9ff-07f6-4f34-9d04-df73378976a5">Original image, noise added, blurred, rotatedHere I have used noise addition, a lowpass filter, and used image rotation to create three new images. These images could indicate disturbances during the measurements, that the camera was out of focus, or that the image was captured using the wrong setting, respectively. If the original label for this image was that the patient was healthy, the labels for the augmented data would be healthy as well.For data augmentation it is important to know the variation that is realistic for our dataset. Synthetic and augmented data is sometimes very useful, but they cannot replace real data, and domain expertise about real data is key in this case!&nbsp;<h2>Closing remarks</h2>Biometric data gives us valuable information about medical conditions and human psychology, and we have not seen the end of the use for this unique and individual data. Having domain experience is crucial when entering the wonderful world of biometric data analysis. To ensure that the ethical requirements are met when we make our analysis, we as data scientists and developers need to keep philosophers and the medical profession close when treading in these deep waters.[1]&nbsp;<a href="https://ec.europa.eu/info/law/law-topic/data-protection/reform/what-personal-data_en#:~:text=Personal%20data%20is%20any%20information,person%2C%20also%20constitute%20personal%20data.">ec.europa.eu</a>[2]&nbsp;<a href="https://commons.wikimedia.org/wiki/Category:Nystagmus#/media/File:Nystagmus_eye_movement.gif">commons.wikimedia.org</a>[3]&nbsp;<a href="https://physionet.org/">https://physionet.org/</a>[4]&nbsp;<a href="https://upload.wikimedia.org/wikipedia/commons/3/34/CT_scan_of_a_pacchionian_body_-_transverse_plane.jpg">CT Scan</a>

Biometric Data & Machine Learning

Marvin who? Marvin from The Hitchhiker’s Guide to the Galaxy of course!

<h2>Yes, you read that right. Building Marvin. Marvin who? Marvin from The Hitchhiker’s Guide to the Galaxy of course!</h2>Let me start by explaining what we built: a computer. And, since we at NordAxon are generally inspired by the Hitchhiker's Guide to the Galaxy, we had to name our power computer after one of the characters.So who is Marvin? For those of you that have not read the books, here comes a tl;dr of the character Marvin the Paranoid Android. In the books, Marvin is one of many failed prototypes of the Sirius Cybernetics Corporation, created to prototype human personality artificial intelligence. Unfortunately, because of his massive brain capacity — allegedly, Marvin is 50 000 times more intelligent than an average human — , he is instead afflicted with severe depression and boredom. No task is too complex for Marvin, no question too deep!And why did NordAxon need to build a computer? The reason why we needed a local power machine here at NordAxon was because sometimes we need to handle sensitive data in our projects, data that we cannot or are not allowed to upload to the cloud. This means that we cannot depend on cloud services when it comes to training machine learning models, and as such, we needed to have our own machine. But why get an already built machine when it is so much more fun to hand-pick all the individual parts of one and building it yourself? So, that is exactly what we did! And Marvin the Paranoid Android, having such computing capacity and being an all-knowing character, made us realize that his name would fit our new power machine perfectly. We imagine that every time we use him for training one of our models, he will become a little bit happier, maybe eventually even so he can leave his depression behind him?No task is too complex for Marvin, no question too deep!&nbsp;<h2>Introducing Marvin our Super Computer</h2><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fmarvin1.jpeg?alt=media&amp;token=ce4d6dc1-a080-4768-a377-1b64110a7c8f">How does one go about building one’s own machine? We will try to explain this in a way that makes this readable both for the novice and the more technical profiles, so hang in there. For those of you that wish to take a look at something even lighter than this article can tune in on our (not at all corny) video about the build, found here!There are in general 8 parts (components) that constitute a computer;<ol><li>GPU (graphics processing unit) — this one makes it possible for you to watch videos, play games and train machine learning models on your computer.</li><li>CPU (central processing unit) — this can be compared to the project leader of the computer; the CPU computes calculations and delegates responsibilities and tasks to the GPU, by using its random access memory (RAM).</li><li>CPU cooler — this component makes sure that the CPU does not overheat when working, and is pretty straight forward; keep the CPU at low temperatures so that it does not break.</li><li>Motherboard — it is via the motherboard that the components communicate to each other, we could compare that to the nerve cells in the body: it is via them the information flows with tasks such as “lift finger”.</li><li>RAM (random access memory) — this is where the CPU stores its temporary variables that it needs to send tasks to other parts of the computer.</li><li>Storage memory — here we have our operating system, together with all of our files and programs</li><li>Power supply —this is what makes sure that all components have the power they need to work, because we all know that nothing fun can happen with no power!</li><li>Case — where we store all of our components, preferably with a nice window to the inside so that we can see and enjoy all the cool components.</li></ol>Don’t worry, I will present some details around the first 4 components and what types of parts we chose and why, starting with the star of the show of course — our GPU.&nbsp;<h2>GPU</h2><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fgpu.jpeg?alt=media&amp;token=69b04f75-2509-4ea4-adb5-d4e74b0ebeb5">Since we work with machine learning (ML) and thus spend a lot of time training ML models, we needed to start our build from a powerful GPU. The reason why this was needed was because GPU:s have the capability to parallelize computing, which means that the training time of machine learning models will be sliced — the better the GPU, the shorter the training time. For those of you that are not that tech-savvy, it could slice training time from days to hours or less (depending on some surrounding variables as well, such as RAM or CPU capability, or training hyperparameters). So, let’s look closer at our choice of GPU.Today, the world leading company when it comes to general purpose GPUs (i.e. GPUs that can be used for other things than strictly graphics processing, such as training machine learning models) is NVIDIA. During fall 2020 they released one of their latest series of GPUs, called the GeForce RTX series, which has proven to be immensely popular. Fortunately for us at NordAxon, we managed to get a hold of the best GPU from this series — the NVIDIA GeForce RTX 3090 with 24 GB memory, more specifically the Gigabyte RTX 3090 24GB TURBO.“But wait, I thought you said NVIDIA and then you wrote Gigabyte?” Yes, yes, we know. It is a bit confusing. Essentially, NVIDIA designs the hardware architecture of graphics cards, and then lets other companies produce them with their own additions to it, such as cooling mechanisms. So, we bought the Gigabyte version with TURBO cooling. Was this Gigabyte Turbo version the only RTX GPU we could get a hold of? Yes. This series has, as we mentioned, been very popular and getting a hold of any GPU in the series is a challenge — they sell out in less than minutes as soon as batches are released. So what does this TURBO cooling mean? Well, essentially the turbo cooling means that the air flow to cool the GPU is set up in a certain way. Now, one of the reasons that we could get a hold of this GPU is because the turbo cooling method is perceived as too weak for a GPU this powerful, meaning that it is possible that the GPU overheats before we reach its full potential — this means that people have been very hesitant to buy this expensive GPU. However, the turbo cooling method is essential if we want to add more GPUs into Marvin in the future, as we otherwise would spew warm air from one GPU to the next, overheating them anyway.Now, is the 3090 the best GPU on the market for training machine learning models? No, of course there are better GPUs, such as the NVIDIA A100 Tensor Core, but for a small startup of 6 employees there is no chance that we could afford one of those. Is the 3090 the best GPU within a reasonable price range for a small startup? Yes! Does Marvin sound like a jet plane due to the loud turbo cooling mechanism? Indeed. So, when we finally had gotten a hold of the GPU, we created the rest of the build. And so, our second component we decided on was the CPU.&nbsp;<h2>CPU</h2>Image Credit: IntelSo for our CPU we chose an Intel Core i9–7900X 3.3 GHz 10-Core Processor. What does all of this mean? Well, the first part, Intel Core i9–7900X, is the name of the CPU, but it is mostly what comes after that is interesting: 3.3 GHz and 10 cores. That first part, 3.3 GHz, is essentially a measure of how many calculations or operations that the CPU can do per time unit. This means at which speed the CPU can compute or assign tasks — the more, the better! The CPU having 10 cores essentially means that we have 10 small CPUs in our big CPU, also enabling parallelized computations — the more, the better applies here as well.And yes, for those of you that keep up to date, we know that this is an old Intel CPU, released already in June 2017. But you know what they say; oldie but goldie.Jokes aside, this CPU choice was a tradeoff. As we wanted to keep the possibility to add more GPUs to our build later on, we were constrained to a choice of motherboards. Given the motherboard that we chose, we were even more constrained regarding CPU, and finally, we went with this one. You have to start somewhere, right?Now, in every build there is going to be a bottleneck. Taking into account that we chose a CPU with “only” 10 cores, this might be it. Have we noticed it bottlenecking us so far? Nope. But technically it could bottleneck us in the future, when we train bigger models with more data.&nbsp;<h2>CPU Cooler</h2><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fcpu.jpeg?alt=media&amp;token=2bc418db-b543-4597-9af9-af59496c9f66">Now for cooling, we realized that an all-in-one (AIO) CPU cooling system probably would do for our CPU, despite its reputation to overheat, and an AIO system looks neat and is easy to install (foreshadowing…). An AIO consists of a pump, two tubes, one radiator and normally two fans mounted on the radiator. We decided on the Corsair H115i PRO 55.4 CFM Liquid CPU Cooler, with subtle RGB on the pump. We know, we know, it is so nerdy with RGB lights in a computer. But we are nerdy at NordAxon... And this cooler gives us the possibility to play around with color schemes during our after works!Now, an AIO cooler works as follows: we place the pump directly over the CPU. Between the pump and the CPU there is something called thermal paste. This paste will conduct heat away from the CPU to the pump. In the pump, and in the entire AIO cooling system, there is a cooling liquid that moves around. The liquid is heated by the CPU, then pumped away into the radiator. The radiator divides the liquid into small sections so that it can be efficiently cooled with fans, and the fans blow air on the radiator.Installing this AIO cooler in the case proved to be a challenge, though. We tried installing it top-mounted, as recommended, but the case turned out to be just a tiny, tiny bit too short for that and the RAMs ended up being in the way. So, there went our air flow planning right out the window and we had to mount radiator and fans at the front of the case, right behind our case fans. So, the radiator got a double set of fans!<h4>There went our air flow planning right out the window and we had to mount radiator and fans at the front of the case</h4>We have not entirely figured out how we want our air flow to be configured for the maximum cooling capacity and the minimum dust intake in the case.Yet another thing for us to ponder during our after works…&nbsp;<h2>Motherboard</h2><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fmotherboard.jpeg?alt=media&amp;token=292b0500-0745-4448-a33c-074889008251">Given the budget we had for Marvin and the potential that we see in him in the near future, we chose the ASUS Pro WS X299 Sage II Workstation Motherboard. Easily put, this motherboard gave us a lot of value for the money and it allows us to add GPUs when needed — it even has fancy-sounding things such as PCIex16 ports, quad-channel memory support and overclocking possibilities. To not make this article unbearable, we will not go into the details of the motherboard more than this, but we will mention that we are fairly happy with this choice!&nbsp;<h2>Building Marvin</h2>So, how did the actual building go? Well, a bit up and down, but everything worked out nice in the end! As mentioned, the AIO radiator and fans were tested on a multitude of places, finally being front mounted behind some fancy RGB case fans. Our data scientists Alexander Hagelborn and Isabella Gagner gladly spent an entire Saturday building Marvin and configuring his software exactly the way they wanted him to work.<img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fmarvin2.jpg?alt=media&amp;token=7cfd7074-e563-4104-9cc5-d663a9f22ff0"><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fmarvin3.jpg?alt=media&amp;token=987b83a9-3fca-491e-8032-67ffe023bf61"><img class="image_resized" style="width:100%;" src="https://ik.imagekit.io/nordaxon040/tr:w-700/v0/b/nordaxon-webapp.appspot.com/o/Blog%2FBuilding%20marvin%2Fmarvin4.jpg?alt=media&amp;token=dac5eb14-d375-4d31-9650-0f574fbdd769">Some photos from the day! Both of us used ESD bracelets — since the room has a wall-to-wall carpet we were worried we might kill components with electrostatic discharge and thus used these bracelets to ground ourselves.And now, as the finale, we will present the different components as seen in the fully built Marvin! The parts that are visible are shown with colored squares and the corresponding part name in the same color. And yes, for those of you that know a thing or two about AIO:s — we will mount the fans and radiator higher to prevent possible air leakage into the system.Do you have any questions about the build or about the performance of Marvin? Let us know by contacting us! We thank you for your time reading about our Marvin adventure and hope you will stay tuned for our next blog posts, that will feature models built and trained on Marvin!

Building The Computer Marvin

By Henrik Siljebråt, Machine Learning Engineer at Nordaxon

Developing tools for taming the black magic of Prompt Engineering & LLMs

A light hearted article on the importance of data quality

This article was originally <a href="https://medium.com/@nordaxon/how-to-fool-a-cnn-in-5-easy-steps-da0fdf0ae7e7">published on medium</a> on the 5 October 2020.Within machine learning, and especially in my team at <a href="https://www.nordaxon.com/">NordAxon</a>, we are constantly talking about the importance of data quality when we are building our models. For beginners, it is easy to forget that we must be mindful of data quality when using our fully trained machine learning models as well — just because we built a model on perfect data (that does not even exist, but in theory!), does not mean that our model can handle any and all challenges coming its way!Let me show you a very pedagogic example of this. This article contains some code for the ones that wish to experiment on their own, but you should be able to follow even if you have no prior programming experience. This is a fun project that I did during work hours, as we in NordAxon believe strongly in having our own projects and continuously educating ourselves — that is our Kaizen!&nbsp;<h2 style="margin-left:0px;">Step 1: Choose a pre-trained model that you wish to fool.</h2>As my data will be images I want to use a Convolutional Neural Network (CNN). The strength in CNNs lie with the convolutional layers, that essentially filter out and boil down important features in each image, all while keeping spatial information! An intuitive way to think about this “spatial information” is that a CNN finds relations between pixels, leading it to understand that a cat consists of two ears at the top, two eyes below, and a nose underneath that — we rarely see cats with the ears below the nose!I will make this very easy and use a pre-trained model from Keras, namely the <a href="https://keras.io/api/applications/vgg/">VGG16 </a>net with ‘imagenet’ weights! For those of you that are not familiar with this concept, it is essentially a fully-trained, pretty big convolutional neural network machine learning model, that has 1000 classes as outputs — this essentially means that the model can recognize 1000 different objects in images. It can classify everything from coffee mugs and cars to elephants and dogs, and it was trained on 1.2 million images from the ImageNet dataset! As I did not have 1.2 million images myself (or the computer power to train such a large network from scratch..), I decided to use the VGG16 and predict on my images directly.&nbsp;<h2 style="margin-left:0px;">Step 2: Download or create images!</h2>Now, let us see what we can use to try to fool this VVG16 network; what could be “dirty data”? As it happens, I stumbled upon <a href="https://www.instagram.com/koty_vezde/?hl=sv">koty_vezde</a>s Instagram account and thought it was perfect. With the approval of the account owner, I have now downloaded 20 images and tried to classify them…<h3 style="margin-left:0px;">So, let us look at the images!</h3><img class="image_resized" style="width:29.18%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_J_MpCTvl7ATC-RTg4Umpwg.jpeg?alt=media&amp;token=73825c07-d9d8-4fb8-9a79-5d01d739f959"> &nbsp;<img class="image_resized" style="width:28.99%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_VHfpvuGAkp8Ncs8O8k9bpg.jpeg?alt=media&amp;token=3b33d6e8-1bab-4689-97c1-9eb759785879"> &nbsp;<img class="image_resized" style="width:28.81%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_CcbbYS00JiucL1mDxYEtDw.jpeg?alt=media&amp;token=bedc3cef-2819-4e49-af5a-deca5c73b51c"><img class="image_resized" style="width:29.23%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_APPJgcmoUHy-jS1u4ebGPA.jpeg?alt=media&amp;token=f505e6a1-8bcf-4ce2-836c-2c32ec03e350"> &nbsp;<img class="image_resized" style="width:19.23%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_iG4qmHXXQFWfho2uH56yvw.jpeg?alt=media&amp;token=52843830-5265-4911-9e65-387cbb14d1f1"> &nbsp;<img class="image_resized" style="width:23.82%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_6_5VKeZpJJPXFOJwOyhveA.jpeg?alt=media&amp;token=975f4280-5110-4b80-8915-4cf2a2a31b1f">Is this not the best thing you have seen today?!&nbsp;<h2 style="margin-left:0px;">Step 3: Enjoy the images for a while</h2>And while you’re at it, take a look on the wonderful <a href="https://www.instagram.com/koty_vezde/?hl=sv">Instagram account</a> where I found these cat fusions! There are a lot of images like these.&nbsp;<h2 style="margin-left:0px;">Step 4: Predict!</h2>This section shows the code for creating the predictions for the images, i.e. what does the model think that it is looking at when presented with these cat-fusion images?Alright, let’s get down to business! Easy peezy lemon squeezy, we read in our images into the shape that our net expects. For our VGG16, that would be tensors of shape (224, 224, 3). That means images that are 224 pixels wide and high, and 3 channels deep for the three colours in RGB.<pre><code class="language-plaintext">import tensorflow as tf
import numpy as np
import os
import keras
from tensorflow.keras.preprocessing import image
from PIL import Image
from tensorflow.keras.applications import VGG16
from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from collections import defaultdict# Read paths into list
dirr = 'Cat-animals'
imgs_paths = [os.path.join(dirr,path) 
 for path in os.listdir(dirr) 
 if path.endswith('jpg')]# Read images and resize them (yes, I do love list comprehensions)
imgs = [image.img_to_array(image.load_img(
 path, target_size=(224, 224))) 
 for path in imgs_paths]# Normalize the colour channels
imgs= np.asarray([preprocess_input(img) for img in imgs])# Print final tensor shape = (20,224,224,3)
print(imgs.shape)</code></pre>Now, we have our images saved in our imgs-tensor. Let’s predict!<pre><code class="language-plaintext">preds = model.predict(imgs)
all_top_3_preds = defaultdict()# Pick the top 3 predictions for each image
for img_pred in range(20):
 top_3_preds = decode_predictions(np.expand_dims(
 preds[img_pred], 
 axis=0), top=3)[0] # Save the top 3 predictions along with the file name
 all_top_3_preds[imgs_path[img_pred]] = [[top_3_preds[i][j+1] 
 for i in range(3) 
 for j in range(2)]]</code></pre><h2 style="margin-left:0px;">&nbsp;</h2><h2 style="margin-left:0px;">Step 5: Evaluate</h2>So, let us see how the VGG16 net classified these images (and remember that the percentages can be interpreted as how sure the model is that the prediction is correct):<img class="image_resized" style="width:30.99%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_rZZImDlHaarG1Gaz7HsgsA.jpeg?alt=media&amp;token=dd5214b6-1076-4c3f-8106-0f499edbfe01"> &nbsp;<img class="image_resized" style="width:31.39%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_-8kw1gMOAcO6kH_unERpSg.jpeg?alt=media&amp;token=2ae4c508-e29d-40c8-99b1-c6278f9658ab"> &nbsp;<img class="image_resized" style="width:31.46%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1__IbQtVC0-eaBlDLLh6COew.jpeg?alt=media&amp;token=8fc28011-9e50-4331-9e92-0df224ca69de"><img class="image_resized" style="width:38.01%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_WGBPMQaON8PufZfPmeX8qQ.jpeg?alt=media&amp;token=88175a14-7d3a-49a9-abf0-aeb53097fc03"> &nbsp;<img class="image_resized" style="width:25.16%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_t7UMjkBYe7Q6K6VJrqFRlA.jpeg?alt=media&amp;token=5c6921aa-f1f7-44b5-a8b7-3551a80bc1e5"> &nbsp;<img class="image_resized" style="width:31.43%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_Kz82FKDreLWmDMGgYFyk6g.jpeg?alt=media&amp;token=f2baa2b1-faf6-4988-86a2-4e99a73bac0d">While it is very easy for us humans to distinguish exactly what has happened in the images, it proved difficult for our dear VGG16 net. None of the top 3 predictions for each image contained “cat”, which actually surprised me! But, when thinking about it a bit more, it makes sense — the model does take a lot of information into consideration when classifying, and while it is easy for us to see “cat” and “squirrel” as two separate objects that have been fused, the model will try to make sense of these things together.In fact, the model assumes that all the data that it sees is correct. It would never dream of us trying to fool it, and as such, it tries to make sense out of all the information that it has. For the model, the majority of the information in most of the pictures point to one animal, or at least one type of animal. The fact that these photoshopped animals are completely out of the realms of reality is not considered, although in some cases the predictions get a bit confused (such as the cat-giraffe, where it sees a cat face and the giraffe pattern and draws the conclusion Cheetah).An important consideration is that the model that you have built, only knows what it has previously seen. And that leads me to my point:It is so important to keep track of the data quality, not only when building your model, but also when using the model. Just because you trained a model on clean data, does not mean that it will perform well on dirty data afterwards. Continuous controls of data, model performance and model verification is of the utmost importance — we cannot have a model that predicts “hummingbird” and “bow tie” where the obvious answers are “fish” and possibly “cat”.&nbsp;<h2 style="margin-left:0px;">Step 6 (optional): Create heatmaps</h2>One of the perks of working with CNNs is that we actually can visualize what the model has activated on when classifying the image. Let us look closer at that!This section is very much inspired by Chollet’s notebook on heatmaps, that you can find on <a href="https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb">this link</a>. This is mostly for fun, and the analysis above persists.Essentially what happens in this code, is that we are looking at the gradients of the convolutional layers in the model. These gradients give us a hint on how much the model looks at the different part of an image when classifying. Intuitively, this can be thought of as “of how much importance is this specific part of the image for the model prediction?”, where the colour red indicates important.<pre><code class="language-plaintext"># We start by looking at the summary of the model. We do this partly
# as a reminder, but also to later get the names of the different 
# layers
model.summary()
layer = 'block5_conv3'# Let us loop through each of the cat images
for i,img_path in enumerate(imgs_paths):
 idx = np.argmax(preds[i])
 # This is the output entry in the prediction vector
 output = model.output[:, idx]
 
 # The is the output feature map of the `block5_conv3` layer,
 # the last convolutional layer in VGG16
 last_conv_layer = model.get_layer(layer)
 
 # This is the gradient of the "african elephant" class with
 # regard to the output feature map of `block5_conv3`
 grads = K.gradients(output, last_conv_layer.output)[0]
 
 # This is a vector of shape (512,), where each entry
 # is the mean intensity of the gradient over a specific feature 
 # map channel
 pooled_grads = K.mean(grads, axis=(0, 1, 2))
 
 # This function allows us to access the values of the 
 # quantities we just defined:
 # `pooled_grads` and the output feature map of `block5_conv3`,
 # given a sample image
 iterate = K.function([model.input], [pooled_grads, 
 last_conv_layer.output[0]])
 
 # These are the values of these two quantities, as Numpy arrays
 # given our sample image of two elephants
 x = imgs[i]
 x = np.expand_dims(x, axis=0)
 pooled_grads_value, conv_layer_output_value = iterate([x])
 
 # We multiply each channel in the feature map array
 # by "how important this channel is" with regard to the elephant
 # class
 for i in range(pooled_grads.shape[0]):
 conv_layer_output_value[:, :, i] *= pooled_grads_value[i]
 
 # The channel-wise mean of the resulting feature map
 # is our heatmap of class activation
 heatmap = np.mean(conv_layer_output_value, axis=-1)
 heatmap = np.maximum(heatmap, 0)
 heatmap /= np.max(heatmap)
 
 # We use cv2 to load the original image, I loaded it as a 
 # gray-scale image as I had a hard time seeing the heatmap 
 # otherwise
 img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)
 img = np.expand_dims(img, axis=2)
 
 # We resize the heatmap to have the same size as the original
 # image
 heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
 # We convert the heatmap to RGB
 heatmap = np.uint8(255 * heatmap) # We apply the heatmap to the original image
 heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET) # 0.5 here is a heatmap intensity factor
 superimposed_img = heatmap * 0.5 + img # Save the image to disk
 cv2.imwrite('Cat-animals/heatmaps/'+
 img_path[12:-4]+'_heatmap_'+
 layer+'.jpg', superimposed_img)</code></pre>&nbsp; &nbsp;&nbsp; <img class="image_resized" style="width:31.96%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_2aZtz7c4Ybr6LQsaAZspDQ.jpeg?alt=media&amp;token=66e2e72b-9d2d-4aa5-8051-1f73a3056c42"> &nbsp;<img class="image_resized" style="width:32.35%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_HJu3eMnv6OTZs2yh1kCmQQ.jpeg?alt=media&amp;token=a173e4df-2bb5-4a6d-9e0b-5afd3f9eb994"> &nbsp;<img class="image_resized" style="width:31.97%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_jlCNG-I9_nuuXgXZCAneAg.jpeg?alt=media&amp;token=43c3ebc5-68dc-4aea-837f-aa72c05997a1"><img class="image_resized" style="width:31.57%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_jT75lBNu58AAqvHoe7V4BQ.jpeg?alt=media&amp;token=7f762ee6-da8b-461d-8442-1bcbc2ce8edf"> &nbsp;<img class="image_resized" style="width:20.79%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_KU0k7DZAmiK-UeedVmKQSQ.jpeg?alt=media&amp;token=55572955-940b-4e65-8085-a93366ece0ac"> &nbsp;<img class="image_resized" style="width:25.84%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_W-nVSVzdU-DYtKEmto0WHg.jpeg?alt=media&amp;token=60560fd0-78d3-4426-abef-60da0dd31823">Alright, now we only looked at the last convolutional layer in the model. If we repeat these steps for each layer, and combine the resulting images into a GIF, we get a better overview of the model and how it activates during the iterations! Let us look at a few.<img class="image_resized" style="width:31.74%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_qu75l_cguTPxofnAejEo3A.gif?alt=media&amp;token=0c187a07-30e0-498b-bf71-9f9555b31b9a"> &nbsp;<img class="image_resized" style="width:21.12%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_1sQzaOnoMVKHAJUPqUvcQw.gif?alt=media&amp;token=eff19d1a-7e85-4f2d-aba0-9dd40de90ca8"> &nbsp;<img class="image_resized" style="width:26.34%;" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/Blog%2F1_Ffp4CGi87hZgm8qpGfW61A.gif?alt=media&amp;token=e23334a4-f31e-4c6c-b8ef-b22ddff4be35">Heatmaps of each of the different layers of the VGG16 Imagenet model.We now see that the model does indeed consider several parts of each image during the layer iterations, starting with edges and moving on to larger areas. The first layers extract local features, such as edges and patterns, and with each layer, more global features and patterns are taken into consideration. At the end, it seems like the model in the majority of the cases discards the “cat-face”-feature as unimportant for the classification, and focuses on the areas around it instead. This shows that the model essentially sees what it wants to see — it does not care that a detail such as the face does not cohere with what is has seen before.Finishing this little project, I was curious of how some of the predictions were wrong even when the model ignored the cat face. As an example, a quick Google image search for Flatworm, that the Octopus was predicted as, shows that the model must have confused the Octopus arms with the sides of a Flatworm— it is possible that this would happen with the original image as well. And the obvious cat faces? Apparently not so obvious for the VGG16 net.<h3 style="margin-left:0px;">Imagine what could happen if a model misclassified objects in a self driving car…</h3>

How to Fool a CNN in 5 Easy Steps

Delving into LLMs' business impact, featuring a NordAxon case study on custom conversational agents and process optimization.

AI & Machine Learning Insights — NordAxon Blog