Facial Emotion Recognition using OpenCV

For this project I tried to use several algorithms to classify emotion based off of images of people with associated emotion labels. The first data set was from the Kaggle Facial Emotion Recognition competition which consisted of 30,000 (48 x 48) images and the second data set was the Cohn-Kanade (CK) database consisting of 700 (350 x 350) images. The main difference between these two data sets is that Cohn-Kanade has the same person exhibiting multiple emotions while Kaggle consists of random people.

I used OpenCV to identify the face in the image and grayscale. When looking at the data at this position, each image is a matrix (350 x 350 for CK and 48 x 48 for Kaggle) with a numerical value for each pixel. After this step, I tried the built in fisher face recognizer from OpenCV to train and test on emotions. I wanted to explore other models as well and so instead of fisher faces, I used Eigenfaces (created through PCA) and fed these vectors into an SVM model, a multilayer perceptron model, and a random forest model. The results are displayed below:

Cohn-Kanade (CK) Kaggle FER2013
SVM 84% 44%
Fisher Face Recognizer 82% 32%
Multilayer Perceptron 77% 37%
RandomForest Classifier 54% 42%

It is clear that models fared much better on the CK data set and this is due to the fact that the model trained on the same person in different emotional categories. Analyzing the same person multiple times made the model more predictive compared to Kaggle where it had to generalize for a completely different set of people in the test data. Using a convolutional neural network increased our accuracy for the Kaggle data set to 57% however (I will cover this network in another blog post).

In order to build a real world application for this I trained the support vector machine model on images of myself and then used my webcam to monitor my emotion. Finally I tied the model to my twitter account so that my current emotion would be tweeted. This toy application is just one example of ways a system like this could be used. I could see mood monitoring being helpful in a psychiatric setting and personally it would be interesting to see how my emotions change as I am performing different tasks at my computer. The gif below shows my implementation and the lag in the appropriate emotion shown is due to the fact that the page is automatically reloading every 5 seconds.


I will go into more depth into the parameter tuning I performed as well as a little bit into using the Twitter API at a later date as an edit. My group also visualized model performance with D3 which I will also cover at a later date. Full code is posted here.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s