iot 4.0
Smart IoT devices
EMOTION RECOGNITION BASED ON SENSORS AND ML
DATASET:
Two EMG channels sensor is located in the Corrugator Supercilii in the forehead's upper part of the eyebrow. This is the primary muscle over the negative emotions of sadness, anger, and disgust (1). Another sensor is allocated in the Zygomaticus major, the muscle located in the cheek near the mouth. This muscle has a high activity on emotions of happiness, fear, and surprise (2). Finally, the last sensor collects data from the Depressor Anuguli Oris, which is in the lower part of the mouth towards the jaw, acting mainly on the emotions of surprise, happiness, and sadness (3).
The dataset can be found in:
- Original samples
- Images
APPROACH:
Human-computer interaction plays a fundamental role in training machine learning models since computers can learn from human behavior. Therefore, this task can be done by cameras and sensors. However, in some scenarios, cameras need a controlled environment to take good-quality pictures and extract patterns from the muscles of the face. Consequently, sensors are a flexible solution in rough applications when humans constantly move or their emotions have weak muscle contractions. In this IoT application, electromyography sensors are allocated to the human face to recognize the six primary emotions: happiness, anger, surprise, fear, sadness, and disgust.
APPROACH A: Merging data from 3 sensors in one sigle array to deploy classification algortihms
APPROACH C: Merging data from 3 sensors in one sigle array to deploy neural networks
APPROACH B: Working with the 3 sensors separately with 100 samples each label to deploy neural networks
READ THE DATASET:
In Python environment:
import numpy as np
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.ndimage import gaussian_filter
#split the dataset
from sklearn.model_selection import train_test_split
# Deep learning
import tensorflow as tf
# Reding the dataset
url = 'https://raw.githubusercontent.com/puldavid87/emotion_recognition/main/emg_data.csv'
data = pd.read_csv(url)
print(data)
EMG1 EMG2 EMG3
0 139.0 113.0 108.0
1 140.0 114.0 108.0
2 142.0 112.0 107.0
3 143.0 116.0 111.0
4 142.0 115.0 114.0
... ... ... ...
29997 117.0 132.0 226.0
29998 115.0 127.0 211.0
29999 113.0 123.0 216.0
[30000 rows x 3 columns]
Function's normalization
def scaler (X):
out = []
min=X.min()
max=X.max()
out=(X-min)/(max-min)
return out
Create the dataset with six labels and 50 samples per label
X=[]
aux = []
sum = 0
for i in range(300):
EMG1 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,0]), sigma=5)
EMG2 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,1]),sigma=5)
EMG3 = gaussian_filter(np.array(data.iloc[0+sum:100+sum,2]),sigma=5)
sum+=100
for j in range(100):
var = [EMG1[j],EMG2[j],EMG3[j]]
var = np.array(var)
aux.append(var)
X.append(aux)
aux = []
X = np.array(X)
X = scaler (X)
print(X.shape)
y = []
for i in range(6):
for j in range(50):
y.append(i)
y=np.array(y)
print(y.shape)
(300, 100, 3)
(300,)
X_con = [np.concatenate(i) for i in X]
X_con = np.array(X_con)
plt.figure(figsize=(10,6))
plt.plot(X_con[269,:])
plt.grid(True)
plt.show()
X_train, X_test, y_train, y_test = train_test_split(X_con, y, test_size = 0.20, random_state=100)
X_train=np.array(X_train)
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
clf = SVC(kernel='linear')
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test,y_pred))
precision recall f1-score support
0 1.00 1.00 1.00 10
1 1.00 1.00 1.00 5
2 0.92 1.00 0.96 11
3 0.93 0.93 0.93 14
4 0.85 0.79 0.81 14
5 0.83 0.83 0.83 6
accuracy 0.92 60
macro avg 0.92 0.92 0.92 60
weighted avg 0.92 0.92 0.92 60
[[10 0 0 0 0 0]
[ 0 5 0 0 0 0]
[ 0 0 11 0 0 0]
[ 0 0 0 13 1 0]
[ 0 0 1 1 11 1]
[ 0 0 0 0 1 5]]
SVM:
from sklearn.neighbors import KNeighborsRegressor
knn_model = KNeighborsRegressor(n_neighbors=3)
knn_model.fit(X_train, y_train)
y_pred = knn_model.predict(X_test)
y_pred=np.round(y_pred)
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test,y_pred))
precision recall f1-score support
0 1.00 0.70 0.82 10
1 0.83 1.00 0.91 5
2 0.77 0.91 0.83 11
3 0.80 0.86 0.83 14
4 0.83 0.71 0.77 14
5 0.71 0.83 0.77 6
accuracy 0.82 60
macro avg 0.83 0.84 0.82 60
weighted avg 0.83 0.82 0.82 60
[[ 7 1 1 1 0 0]
[ 0 5 0 0 0 0]
[ 0 0 10 1 0 0]
[ 0 0 1 12 1 0]
[ 0 0 1 1 10 2]
[ 0 0 0 0 1 5]]
kNN:
samples=X[0:6,:,:]
fig, axs = plt.subplots(3, 2, figsize=(20, 8), constrained_layout=True)
ax1 = plt.subplot(231)
ax1.plot(samples[0,:,0], 'b', label='EMG1')
ax1.plot(samples[0,:,1], 'g', label='EMG1')
ax1.plot(samples[0,:,2], 'r', label='EMG1')
ax1.grid(True)
ax2 = plt.subplot(232)
ax2.plot(samples[1,:,0], 'b', label='EMG1')
ax2.plot(samples[1,:,1], 'g', label='EMG1')
ax2.plot(samples[1,:,2], 'r', label='EMG1')
ax2.grid(True)
ax3 = plt.subplot(233)
ax3.plot(samples[2,:,0], 'b', label='EMG1')
ax3.plot(samples[2,:,1], 'g', label='EMG1')
ax3.plot(samples[2,:,2], 'r', label='EMG1')
ax3.grid(True)
ax4 = plt.subplot(234)
ax4.plot(samples[3,:,0], 'b', label='EMG1')
ax4.plot(samples[3,:,1], 'g', label='EMG1')
ax4.plot(samples[3,:,2], 'r', label='EMG1')
ax4.grid(True)
ax5 = plt.subplot(235)
ax5.plot(samples[4,:,0], 'b', label='EMG1')
ax5.plot(samples[4,:,1], 'g', label='EMG1')
ax5.plot(samples[4,:,2], 'r', label='EMG1')
ax5.grid(True)
ax6 = plt.subplot(236)
ax6.plot(samples[5,:,0], 'b', label='EMG1')
ax6.plot(samples[5,:,1], 'g', label='EMG1')
ax6.plot(samples[5,:,2], 'r', label='EMG1')
ax6.grid(True)
plt.show()
Plot samples
Split the dataset in train and test set
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier()
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test,y_pred))
Decision Tree:
precision recall f1-score support
0 1.00 1.00 1.00 10
1 0.83 1.00 0.91 5
2 0.91 0.91 0.91 11
3 1.00 0.93 0.96 14
4 1.00 0.86 0.92 14
5 0.75 1.00 0.86 6
accuracy 0.93 60
macro avg 0.92 0.95 0.93 60
weighted avg 0.94 0.93 0.93 60
[[ 10 0 0 0 0 0]
[ 0 5 0 0 0 0]
[ 0 0 10 0 0 1]
[ 0 0 1 13 0 0]
[ 0 1 0 0 12 1]
[ 0 0 0 0 0 6]]
Naive Bayes:
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
y_pred = gnb.fit(X_train, y_train).predict(X_test)
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test,y_pred))
precision recall f1-score support
0 1.00 1.00 1.00 10
1 0.83 1.00 0.91 5
2 1.00 1.00 1.00 11
3 1.00 1.00 1.00 14
4 1.00 0.86 0.92 14
5 0.86 1.00 0.92 6
accuracy 0.97 60
macro avg 0.95 0.98 0.96 60
weighted avg 0.97 0.97 0.97 60
[[ 10 0 0 0 0 0]
[ 0 5 0 0 0 0]
[ 0 0 11 0 0 0]
[ 0 0 0 14 0 0]
[ 0 1 0 0 12 1]
[ 0 0 0 0 0 6]]
Convert labels in categorical
y = []
for i in range(6):
for j in range(50):
y.append(i)
y=np.array(y)
y.shape
y=tf.keras.utils.to_categorical(y,6)
print(y)
[[1. 0. 0. 0. 0. 0.]
[1. 0. 0. 0. 0. 0.]
[1. 0. 0. 0. 0. 0.]
...
[0. 0. 0. 0. 0. 1.]
[0. 0. 0. 0. 0. 1.]
[0. 0. 0. 0. 0. 1.]]
X_train, X_test, Y_train, Y_test = train_test_split(X,y, test_size = 0.10, random_state = 42)
X_train = X_train.reshape(270,100,3,1)
X_test = X_test.reshape(30,100,3,1)
print(X_train.shape,Y_train.shape)
print(X_test.shape,Y_test.shape)
Split the dataset in train and test set
(270, 100, 3, 1) (270, 6)
(30, 100, 3, 1) (30, 6)
Define the neural network model
X_train, X_test, Y_train, Y_test = train_test_split(X,y, test_size = 0.10, random_state = 42)
X_train = X_train.reshape(270,100,3,1)
X_test = X_test.reshape(30,100,3,1)
print(X_train.shape,Y_train.shape)
print(X_test.shape,Y_test.shape)
Model: "sequential" _________________________________________________________________
Layer (type) Output Shape Param # =================================================================
Input (Dense) (None, 100, 80) 320
layer1 (Dense) (None, 100, 40) 3240
layer2 (Dense) (None, 100, 20) 820
flatten (Flatten) (None, 2000) 0
output (Dense) (None, 6) 12006 =================================================================
Total params: 16,386
Trainable params: 16,386
Non-trainable params: 0 _________________________________________________________________
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
nn = Sequential()
nn.add(Dense(units = 80, activation="relu", input_shape = (100,3), name='Input'))
nn.add(Dense(units = 40, activation="relu", name='layer1'))
nn.add(Dense(units = 20, activation="relu" , name='layer2'))
nn.add(Flatten())
nn.add(Dense(units = 6, activation='softmax',name='output'))
nn.summary()
nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history=nn.fit(X_train, Y_train, epochs = 20, validation_data=(X_test, Y_test), batch_size = 20)
Epoch 1/20
14/14 [==============================] - 1s 18ms/step - loss: 1.7033 - accuracy: 0.2556 - val_loss: 1.5782 - val_accuracy: 0.4333 Epoch 2/20
14/14 [==============================] - 0s 5ms/step - loss: 1.3851 - accuracy: 0.5556 - val_loss: 1.1691 - val_accuracy: 0.6667
. . .
Epoch 18/20
14/14 [==============================] - 0s 6ms/step - loss: 0.0607 - accuracy: 0.9889 - val_loss: 0.1612 - val_accuracy: 0.9000 Epoch 19/20
14/14 [==============================] - 0s 5ms/step - loss: 0.0653 - accuracy: 0.9815 - val_loss: 0.2021 - val_accuracy: 0.9000 Epoch 20/20
14/14 [==============================] - 0s 5ms/step - loss: 0.0663 - accuracy: 0.9926 - val_loss: 0.0983 - val_accuracy: 1.0000
y_pred = nn.predict(X_test)
y_pred = (y_pred > 0.5)
index=np.argmax(y_pred)
index
from sklearn import metrics
print("")
print("Precision: {}%".format(100*metrics.precision_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("Recall: {}%".format(100*metrics.recall_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("f1_score: {}%".format(100*metrics.f1_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("Error: {}%".format(metrics.mean_absolute_error(Y_test.argmax(axis=1),y_pred.argmax(axis=1))))
from sklearn.metrics import confusion_matrix, accuracy_score,classification_report
report=classification_report(Y_test.argmax(axis=1),y_pred.argmax(axis=1))
print('\Report\n')
print(report)
precision recall f1-score support
0 1.00 1.00 1.00 10
1 1.00 1.00 1.00 5
2 1.00 1.00 1.00 11
3 1.00 1.00 1.00 14
4 1.00 1.00 1.00 14
5 1.00 1.00 1.00 6
accuracy 1.00 60
macro avg 1.00 1.00 1.00 60
weighted avg 1.00 1.00 1.00 60
X_train, X_test, Y_train, Y_test = train_test_split(X_con,y, test_size = 0.10, random_state = 42)
X_train = X_train.reshape(270,300,1)
X_test = X_test.reshape(30,300,1)
print(X_train.shape,Y_train.shape)
print(X_test.shape,Y_test.shape)
(270, 300, 1) (270, 6)
(30, 300, 1) (30, 6)
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
nn = Sequential()
nn.add(Dense(units = 100, activation="relu", input_shape = (100), name='Input'))
nn.add(Dense(units = 50, activation="relu", name='layer1'))
nn.add(Dense(units = 25, activation="relu" , name='layer2'))
nn.add(Flatten())
nn.add(Dense(units = 6, activation='softmax',name='output'))
nn.summary()
Model: "sequential" _________________________________________________________________
Layer (type) Output Shape Param # =================================================================
Input (Dense) (None, 100) 30100
layer1 (Dense) (None,50) 5050
layer2 (Dense) (None, 25) 1275
flatten (Flatten) (None, 25) 0
output (Dense) (None, 6) 156 =================================================================
Total params: 36581
Trainable params: 36581
Non-trainable params: 0 _________________________________________________________________
Define the neural network model
nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history=nn.fit(X_train, Y_train, epochs = 20, validation_data=(X_test, Y_test), batch_size = 10)
Epoch 1/20
27/27 [==============================] - 0s 6ms/step - loss: 1.5653 - accuracy: 0.4630 - val_loss: 1.4552 - val_accuracy: 0.4667 Epoch 2/20
27/27 [==============================] - 0s 2ms/step - loss: 1.0780 - accuracy: 0.6704 - val_loss: 0.9394 - val_accuracy: 0.6667
. . .
Epoch 18/20
27/27 [==============================] - 0s 2ms/step - loss: 0.0816 - accuracy: 0.9741 - val_loss: 0.2282 - val_accuracy: 0.8667 Epoch 19/20
27/27 [==============================] - 0s 2ms/step - loss: 0.0813 - accuracy: 0.9741 - val_loss: 0.1765 - val_accuracy: 0.9333 Epoch 20/20
27/27 [==============================] - 0s 2ms/step - loss: 0.0868 - accuracy: 0.9667 - val_loss: 0.1367 - val_accuracy: 0.9333
y_pred = nn.predict(X_test)
y_pred = (y_pred > 0.5)
index=np.argmax(y_pred)
index
from sklearn import metrics
print("")
print("Precision: {}%".format(100*metrics.precision_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("Recall: {}%".format(100*metrics.recall_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("f1_score: {}%".format(100*metrics.f1_score(Y_test.argmax(axis=1),y_pred.argmax(axis=1), average="weighted")))
print("Error: {}%".format(metrics.mean_absolute_error(Y_test.argmax(axis=1),y_pred.argmax(axis=1))))
from sklearn.metrics import confusion_matrix, accuracy_score,classification_report
report=classification_report(Y_test.argmax(axis=1),y_pred.argmax(axis=1))
print('\Report\n')
print(report)
precision recall f1-score support
0 1.00 1.00 1.00 4
1 1.00 1.00 1.00 2
2 1.00 1.00 1.00 4
3 0.86 1.00 0.92 6
4 1.00 0.78 0.88 9
5 0.83 1.00 0.91 5
accuracy 0.93 30
macro avg 0.95 0.96 0.95 30
weighted avg 0.94 0.93 0.93 30
RESOURCES:
- Python code, ML models, and the dataset are available in: https://github.com/puldavid87/emotion_recognition
- Article (draft version)
EXPORTING MODELS:
Decision Tree:
#!pip install m2cgen #run once
import m2cgen as m2c
decision_tree_model=m2c.export_to_c(tree)
print(decision_tree_model)
with open('tree.h', 'w') as file:
file.write(decision_tree_model)
Neural Network
More information in: https://www.tensorflow.org/lite/microcontrollers/get_started_low_level
# Define paths to model files
import os
MODELS_DIR = 'models/'
if not os.path.exists(MODELS_DIR):
os.mkdir(MODELS_DIR)
MODEL_TF = MODELS_DIR + 'model'
MODEL_NO_QUANT_TFLITE = MODELS_DIR + 'model_no_quant.tflite'
MODEL_QUANT_TFLITE= MODELS_DIR+ 'model_quant.tflite'
MODEL_TFLITE = MODELS_DIR + 'model.tflite'
MODEL_TFLITE_MICRO = MODELS_DIR + 'model.cc'
# Save the model to disk
nn.save(MODEL_TF)
# Convert the model to the TensorFlow Lite format without quantization
converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_TF)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
model_no_quant_tflite = converter.convert()
# Save the model to disk
open(MODEL_NO_QUANT_TFLITE, "wb").write(model_no_quant_tflite)
# Convert the model to the TensorFlow Lite format with quantization
def representative_dataset():
yield([X_train[0:50,:,:,:].astype(np.float32)])
converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_TF)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
model_quant_tflite = converter.convert()
# Save the model to disk
open(MODEL_QUANT_TFLITE, "wb").write(model_quant_tflite)
# Set the optimization flag.
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Enforce integer only quantization
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
# Provide a representative dataset to ensure we quantize correctly.
converter.representative_dataset = representative_dataset
model_tflite = converter.convert()
# Save the model to disk
open(MODEL_TFLITE, "wb").write(model_tflite)
CONCLUSIONS:
- The neural network model was quantized to be compiled into the microcontroller to reduce sending information to the cloud. Therefore, the electronic system is portable and can send information to computers with low computational resources since the model does not need to be updated.
- The ML pipeline shows an adequate data collection step with the correct allocation of EMG sensors on the face using the facial actions coding system, which describes the specific facial muscle activation
- The deep learning model can correctly recognize over 92\% of human emotions in women and men. Besides, it demonstrated that artificial emotions without stimuli could be detected with sensors and showed that people's natural facial expressions are standardized.
