EMOTION RECOGNITION BASED ON SENSORS AND ML

DATASET: 

Two EMG channels sensor is located in the Corrugator Supercilii in the forehead's upper part of the eyebrow. This is the primary muscle over the negative emotions of sadness, anger, and disgust (1). Another sensor is allocated in the Zygomaticus major, the muscle located in the cheek near the mouth. This muscle has a high activity on emotions of happiness, fear, and surprise (2). Finally, the last sensor collects data from the Depressor Anuguli Oris, which is in the lower part of the mouth towards the jaw, acting mainly on the emotions of surprise, happiness, and sadness (3). 

 

The dataset can be found in:

  • Original samples
  • Images

APPROACH:

Human-computer interaction plays a fundamental role in training machine learning models since computers can learn from human behavior. Therefore, this task can be done by cameras and sensors. However, in some scenarios, cameras need a controlled environment to take good-quality pictures and extract patterns from the muscles of the face. Consequently, sensors are a flexible solution in rough applications when humans constantly move or their emotions have weak muscle contractions. In this IoT application,  electromyography sensors are allocated to the human face to recognize the six primary emotions: happiness, anger, surprise, fear, sadness, and disgust. 

APPROACH A: Merging data from 3 sensors in one sigle array to deploy classification algortihms

APPROACH C: Merging data from 3 sensors in one sigle array to deploy neural networks

APPROACH B: Working with the 3 sensors separately with 100 samples each label to deploy neural networks

READ THE DATASET:

In Python environment:

 

import numpy as np

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from scipy.ndimage import gaussian_filter

#split the dataset

from sklearn.model_selection import train_test_split

# Deep learning

import tensorflow as tf

 

# Reding the dataset

url = 'https://raw.githubusercontent.com/puldavid87/emotion_recognition/main/emg_data.csv'

data = pd.read_csv(url)

print(data)

       EMG1  EMG2  EMG3

    0  139.0 113.0 108.0

    1  140.0 114.0 108.0

    2  142.0 112.0 107.0

    3  143.0 116.0 111.0

    4  142.0 115.0 114.0

   ... ...   ...   ...

 29997 117.0 132.0 226.0 

 29998 115.0 127.0 211.0

 29999 113.0 123.0 216.0

 [30000 rows x 3 columns]

Function's normalization

def scaler (X):

  out = []

  min=X.min()

  max=X.max()

  out=(X-min)/(max-min)

  return out

Create the dataset with six labels and 50 samples per label

X=[]

aux = []

sum = 0

for i in range(300):

  EMG1 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,0]), sigma=5)

  EMG2 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,1]),sigma=5)

  EMG3 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,2]),sigma=5)  

  sum+=100

  for j in range(100):

    var = [EMG1[j],EMG2[j],EMG3[j]]

    var = np.array(var)

    aux.append(var)

  X.append(aux)

  aux = []

X = np.array(X)

X = scaler (X) 

print(X.shape)

 

y = []

for i in range(6):

  for j in range(50):

    y.append(i)

y=np.array(y)

print(y.shape)

(300, 100, 3)

(300,)

X_con = [np.concatenate(i) for i in X]

X_con = np.array(X_con)

plt.figure(figsize=(10,6))

plt.plot(X_con[269,:])

plt.grid(True)

plt.show()

X_train, X_test, y_train, y_test = train_test_split(X_con, y, test_size = 0.20, random_state=100)

X_train=np.array(X_train)

from sklearn.svm import SVC

from sklearn.model_selection import train_test_split

from sklearn import metrics

from sklearn.metrics import confusion_matrix

from sklearn.metrics import classification_report

 

clf = SVC(kernel='linear')

clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)

print(classification_report(y_test, y_pred))

print(confusion_matrix(y_test,y_pred))

 

             precision recall f1-score support

           0   1.00     1.00   1.00      10

           1   1.00     1.00   1.00       5

           2   0.92     1.00   0.96      11

           3   0.93     0.93   0.93      14

           4   0.85     0.79   0.81      14

           5   0.83     0.83   0.83       6

accuracy                       0.92      60

macro avg      0.92     0.92   0.92      60

weighted avg   0.92     0.92   0.92      60

[[10 0 0 0 0 0]

 [ 0 5 0 0 0 0]

 [ 0 0 11 0 0 0]

 [ 0 0 0 13 1 0]

 [ 0 0 1 1 11 1]

 [ 0 0 0 0 1 5]]

SVM:

from sklearn.neighbors import KNeighborsRegressor

knn_model = KNeighborsRegressor(n_neighbors=3)

knn_model.fit(X_train, y_train)

y_pred = knn_model.predict(X_test)

y_pred=np.round(y_pred)

print(classification_report(y_test, y_pred))

print(confusion_matrix(y_test,y_pred))

 

             precision  recall  f1-score support

          0    1.00       0.70     0.82       10

          1    0.83       1.00     0.91        5

          2    0.77       0.91     0.83       11

          3    0.80       0.86     0.83       14

          4    0.83       0.71     0.77       14

          5    0.71       0.83     0.77        6

   accuracy                        0.82       60

  macro avg    0.83       0.84     0.82       60

weighted avg   0.83       0.82     0.82       60

[[ 7 1 1 1 0 0]

 [ 0 5 0 0 0 0]

 [ 0 0 10 1 0 0]

 [ 0 0 1 12 1 0]

 [ 0 0 1 1 10 2]

 [ 0 0 0 0 1 5]]

kNN:

samples=X[0:6,:,:]

fig, axs = plt.subplots(3, 2, figsize=(20, 8), constrained_layout=True)

ax1 = plt.subplot(231)

ax1.plot(samples[0,:,0],  'b', label='EMG1')

ax1.plot(samples[0,:,1],  'g', label='EMG1')

ax1.plot(samples[0,:,2],  'r', label='EMG1')

ax1.grid(True)

 

ax2 = plt.subplot(232)

ax2.plot(samples[1,:,0],  'b', label='EMG1')

ax2.plot(samples[1,:,1],  'g', label='EMG1')

ax2.plot(samples[1,:,2],  'r', label='EMG1')

ax2.grid(True)

 

ax3 = plt.subplot(233)

ax3.plot(samples[2,:,0],  'b', label='EMG1')

ax3.plot(samples[2,:,1],  'g', label='EMG1')

ax3.plot(samples[2,:,2],  'r', label='EMG1')

ax3.grid(True)

 

ax4 = plt.subplot(234)

ax4.plot(samples[3,:,0],  'b', label='EMG1')

ax4.plot(samples[3,:,1],  'g', label='EMG1')

ax4.plot(samples[3,:,2],  'r', label='EMG1')

ax4.grid(True)

 

ax5 = plt.subplot(235)

ax5.plot(samples[4,:,0],  'b', label='EMG1')

ax5.plot(samples[4,:,1],  'g', label='EMG1')

ax5.plot(samples[4,:,2],  'r', label='EMG1')

ax5.grid(True)

 

ax6 = plt.subplot(236)

ax6.plot(samples[5,:,0],  'b', label='EMG1')

ax6.plot(samples[5,:,1],  'g', label='EMG1')

ax6.plot(samples[5,:,2],  'r', label='EMG1')

ax6.grid(True)

plt.show()

  

Plot samples

Split the dataset in train and test set

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier()

tree.fit(X_train, y_train)

y_pred = tree.predict(X_test)

print(classification_report(y_test, y_pred))

print(confusion_matrix(y_test,y_pred))

Decision Tree:

             precision  recall  f1-score support

          0    1.00       1.00     1.00       10

          1    0.83       1.00     0.91        5

          2    0.91       0.91     0.91       11

          3    1.00       0.93     0.96       14

          4    1.00       0.86     0.92       14

          5    0.75       1.00     0.86        6

   accuracy                        0.93       60

  macro avg    0.92       0.95     0.93       60

weighted avg   0.94       0.93     0.93       60

[[ 10 0 0 0 0 0]

 [ 0 5 0 0 0 0]

 [ 0 0 10 0 0 1]

 [ 0 0 1 13 0 0]

 [ 0 1 0 0 12 1]

 [ 0 0 0 0 0 6]]

Naive Bayes:

from sklearn.naive_bayes import GaussianNB

gnb = GaussianNB()

y_pred = gnb.fit(X_train, y_train).predict(X_test)

print(classification_report(y_test, y_pred))

print(confusion_matrix(y_test,y_pred))

             precision  recall  f1-score support

          0    1.00       1.00     1.00       10

          1    0.83       1.00     0.91        5

          2    1.00       1.00     1.00       11

          3    1.00       1.00     1.00       14

          4    1.00       0.86     0.92       14

          5    0.86       1.00     0.92        6

   accuracy                        0.97       60

  macro avg    0.95       0.98     0.96       60

weighted avg   0.97       0.97     0.97       60

[[ 10 0 0 0 0 0]

 [ 0 5 0 0 0 0]

 [ 0 0 11 0 0 0]

 [ 0 0 0 14 0 0]

 [ 0 1 0 0 12 1]

 [ 0 0 0 0 0 6]]

Convert labels in categorical

y = []

for i in range(6):

  for j in range(50):

    y.append(i)

y=np.array(y)

y.shape

y=tf.keras.utils.to_categorical(y,6)

print(y)

[[1. 0. 0. 0. 0. 0.]

 [1. 0. 0. 0. 0. 0.]

 [1. 0. 0. 0. 0. 0.]

 ...

 [0. 0. 0. 0. 0. 1.]

 [0. 0. 0. 0. 0. 1.]

 [0. 0. 0. 0. 0. 1.]]

X_train, X_test, Y_train, Y_test = train_test_split(X,y, test_size = 0.10, random_state = 42)

X_train = X_train.reshape(270,100,3,1)

X_test = X_test.reshape(30,100,3,1)

print(X_train.shape,Y_train.shape)

print(X_test.shape,Y_test.shape)

Split the dataset in train and test set

(270, 100, 3, 1) (270, 6)

(30, 100, 3, 1) (30, 6)

Define the neural network model

X_train, X_test, Y_train, Y_test = train_test_split(X,y, test_size = 0.10, random_state = 42)

X_train = X_train.reshape(270,100,3,1)

X_test = X_test.reshape(30,100,3,1)

print(X_train.shape,Y_train.shape)

print(X_test.shape,Y_test.shape)

Model: "sequential" _________________________________________________________________

Layer (type)              Output Shape                 Param # =================================================================

Input (Dense)          (None, 100, 80)                    320

layer1 (Dense)         (None, 100, 40)                   3240

layer2 (Dense)         (None, 100, 20)                    820

flatten (Flatten)      (None, 2000)                         0

output (Dense)         (None, 6)                        12006 =================================================================

Total params: 16,386

Trainable params: 16,386

Non-trainable params: 0 _________________________________________________________________

from keras.models import Sequential

from keras.layers import Dense

from keras.layers import Flatten

 

nn = Sequential()

nn.add(Dense(units = 80, activation="relu", input_shape = (100,3), name='Input'))

nn.add(Dense(units = 40, activation="relu", name='layer1'))

nn.add(Dense(units = 20, activation="relu" , name='layer2'))

nn.add(Flatten())

nn.add(Dense(units = 6, activation='softmax',name='output'))

nn.summary()

nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history=nn.fit(X_train, Y_train, epochs = 20, validation_data=(X_test, Y_test), batch_size = 20)

Epoch 1/20

14/14 [==============================] - 1s 18ms/step - loss: 1.7033 - accuracy: 0.2556 - val_loss: 1.5782 - val_accuracy: 0.4333 Epoch 2/20

14/14 [==============================] - 0s 5ms/step - loss: 1.3851 - accuracy: 0.5556 - val_loss: 1.1691 - val_accuracy: 0.6667

. . .

Epoch 18/20

14/14 [==============================] - 0s 6ms/step - loss: 0.0607 - accuracy: 0.9889 - val_loss: 0.1612 - val_accuracy: 0.9000 Epoch 19/20

14/14 [==============================] - 0s 5ms/step - loss: 0.0653 - accuracy: 0.9815 - val_loss: 0.2021 - val_accuracy: 0.9000 Epoch 20/20

14/14 [==============================] - 0s 5ms/step - loss: 0.0663 - accuracy: 0.9926 - val_loss: 0.0983 - val_accuracy: 1.0000

y_pred = nn.predict(X_test)

y_pred = (y_pred > 0.5)

index=np.argmax(y_pred)

index

from sklearn import metrics

print("")

print("Precision: {}%".format(100*metrics.precision_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("Recall: {}%".format(100*metrics.recall_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("f1_score: {}%".format(100*metrics.f1_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("Error: {}%".format(metrics.mean_absolute_error(Y_test.argmax(axis=1),y_pred.argmax(axis=1))))

from sklearn.metrics import confusion_matrix, accuracy_score,classification_report

report=classification_report(Y_test.argmax(axis=1),y_pred.argmax(axis=1))

print('\Report\n')

print(report)

             precision  recall  f1-score support

          0    1.00       1.00     1.00       10

          1    1.00       1.00     1.00        5

          2    1.00       1.00     1.00       11

          3    1.00       1.00     1.00       14

          4    1.00       1.00     1.00       14

          5    1.00       1.00     1.00        6

   accuracy                        1.00       60

  macro avg    1.00       1.00     1.00       60

weighted avg   1.00       1.00     1.00       60

X_train, X_test, Y_train, Y_test = train_test_split(X_con,y, test_size = 0.10, random_state = 42)

X_train = X_train.reshape(270,300,1)

X_test = X_test.reshape(30,300,1)

print(X_train.shape,Y_train.shape)

print(X_test.shape,Y_test.shape)

(270, 300, 1) (270, 6)

(30, 300, 1) (30, 6)

from keras.models import Sequential

from keras.layers import Dense

from keras.layers import Flatten

 

nn = Sequential()

nn.add(Dense(units = 100, activation="relu", input_shape = (100), name='Input'))

nn.add(Dense(units = 50, activation="relu", name='layer1'))

nn.add(Dense(units = 25, activation="relu" , name='layer2'))

nn.add(Flatten())

nn.add(Dense(units = 6, activation='softmax',name='output'))

nn.summary()

Model: "sequential" _________________________________________________________________

Layer (type)              Output Shape                 Param # =================================================================

Input (Dense)          (None, 100)                       30100

layer1 (Dense)         (None,50)                          5050

layer2 (Dense)         (None, 25)                         1275

flatten (Flatten)      (None, 25)                            0

output (Dense)         (None, 6)                           156            =================================================================

Total params: 36581

Trainable params: 36581

Non-trainable params: 0 _________________________________________________________________

Define the neural network model

nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history=nn.fit(X_train, Y_train, epochs = 20, validation_data=(X_test, Y_test), batch_size = 10)

Epoch 1/20

27/27 [==============================] - 0s 6ms/step - loss: 1.5653 - accuracy: 0.4630 - val_loss: 1.4552 - val_accuracy: 0.4667 Epoch 2/20

27/27 [==============================] - 0s 2ms/step - loss: 1.0780 - accuracy: 0.6704 - val_loss: 0.9394 - val_accuracy: 0.6667

. . .

Epoch 18/20

27/27 [==============================] - 0s 2ms/step - loss: 0.0816 - accuracy: 0.9741 - val_loss: 0.2282 - val_accuracy: 0.8667 Epoch 19/20

27/27 [==============================] - 0s 2ms/step - loss: 0.0813 - accuracy: 0.9741 - val_loss: 0.1765 - val_accuracy: 0.9333 Epoch 20/20

27/27 [==============================] - 0s 2ms/step - loss: 0.0868 - accuracy: 0.9667 - val_loss: 0.1367 - val_accuracy: 0.9333

y_pred = nn.predict(X_test)

y_pred = (y_pred > 0.5)

index=np.argmax(y_pred)

index

from sklearn import metrics

print("")

print("Precision: {}%".format(100*metrics.precision_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("Recall: {}%".format(100*metrics.recall_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("f1_score: {}%".format(100*metrics.f1_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))

print("Error: {}%".format(metrics.mean_absolute_error(Y_test.argmax(axis=1),y_pred.argmax(axis=1))))

from sklearn.metrics import confusion_matrix, accuracy_score,classification_report

report=classification_report(Y_test.argmax(axis=1),y_pred.argmax(axis=1))

print('\Report\n')

print(report)

             precision  recall  f1-score support

          0    1.00       1.00     1.00        4

          1    1.00       1.00     1.00        2

          2    1.00       1.00     1.00        4

          3    0.86       1.00     0.92        6

          4    1.00       0.78     0.88        9

          5    0.83       1.00     0.91        5

   accuracy                        0.93       30

  macro avg    0.95       0.96     0.95       30

weighted avg   0.94       0.93     0.93       30

RESOURCES:

EXPORTING MODELS:

 

Decision Tree:

 

#!pip install m2cgen  #run once

import m2cgen as m2c

 

decision_tree_model=m2c.export_to_c(tree)

print(decision_tree_model)

 

with open('tree.h', 'w') as file:

    file.write(decision_tree_model)

# Define paths to model files

import os

MODELS_DIR = 'models/'

if not os.path.exists(MODELS_DIR):

    os.mkdir(MODELS_DIR)

MODEL_TF = MODELS_DIR + 'model'

MODEL_NO_QUANT_TFLITE = MODELS_DIR + 'model_no_quant.tflite'

MODEL_QUANT_TFLITE= MODELS_DIR+ 'model_quant.tflite'

MODEL_TFLITE = MODELS_DIR + 'model.tflite'

MODEL_TFLITE_MICRO = MODELS_DIR + 'model.cc'

 

# Save the model to disk

nn.save(MODEL_TF)

 

# Convert the model to the TensorFlow Lite format without quantization

converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_TF)

converter.optimizations = [tf.lite.Optimize.DEFAULT]

model_no_quant_tflite = converter.convert()

 

# Save the model to disk

open(MODEL_NO_QUANT_TFLITE, "wb").write(model_no_quant_tflite)

 

# Convert the model to the TensorFlow Lite format with quantization

def representative_dataset():

     yield([X_train[0:50,:,:,:].astype(np.float32)])

 

converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_TF)

converter.optimizations = [tf.lite.Optimize.DEFAULT]

converter.representative_dataset = representative_dataset

model_quant_tflite = converter.convert()

# Save the model to disk

open(MODEL_QUANT_TFLITE, "wb").write(model_quant_tflite)

# Set the optimization flag.

converter.optimizations = [tf.lite.Optimize.DEFAULT]

# Enforce integer only quantization

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]

converter.inference_input_type = tf.int8

converter.inference_output_type = tf.int8

# Provide a representative dataset to ensure we quantize correctly.

converter.representative_dataset = representative_dataset

model_tflite = converter.convert()

 

# Save the model to disk

open(MODEL_TFLITE, "wb").write(model_tflite)

CONCLUSIONS:

  • The neural network model was quantized to be compiled into the microcontroller to reduce sending information to the cloud. Therefore, the electronic system is portable and can send information to computers with low computational resources since the model does not need to be updated.
  •  The ML pipeline shows an adequate data collection step with the correct allocation of EMG sensors on the face using the facial actions coding system, which describes the specific facial muscle activation
  • The deep learning model can correctly recognize over 92\% of human emotions in women and men. Besides, it demonstrated that artificial emotions without stimuli could be detected with sensors and showed that people's natural facial expressions are standardized.