The non-contact biometric identified bio signal measurement sensor and algorithms

2.1Face position detection using PCA (principal component analysis)

Form Eigen face (common information of the facial types) using PCA, then specify the part that has the similar form to the input image as a person’s face, and track it. PCA is a secondary statistical method using the statistical nature to the mean and variance. For input data, PCA finds a set of orthogonal normalized axes, which points in each direction of the maximum normalized covariance with respect to the input data. This has the advantage of reducing the dimensions of the material efficiently by finding the most important axes of input data. However, PCA is difficult to show the most basic features in the video because it only utilizes secondary statistics.

Given data X, and if there are n pieces of observed samples, then X can be defined as X=[x1,x2,…,xn]. Here, each sample xi of X consists of m pieces of data like xi=[xi⁢(1),xi⁢(2),…,xi⁢(m)]T constituting the sample. Here, T denotes the transposition of a matrix, and in case of the face image, m value becomes the number of pixels in the image of the face and can be expressed as a one-dimensional vector.

The method for representing data by PCA is as follows. R represents data by PCA, and at this time, each row is matched to the sample of the original data. When a matrix containing eigenvectors is arranged in columns it named V. Then, R can be obtained by R=X⁢T⁢V. So, it has the properties of V⁢V⁢T= 1 since eigenvector V is symmetrical and orthogonally normalized. The conversion of the data in the reverse can be obtained as X⁢T=R⁢V⁢T.

The feature vector space obtained by applying PCA results in the recognition rate falls because it includes the features such as changes in illumination of the image and in facial expression. Considering this, it only takes the approximate location of the face through the PCA and extracts the feature points in that location. We intend to improve the recognition rate by recognizing the face using the comparison of the feature points.

2.2The feature point extraction in the detected face position

Since a face recognition method usually uses the entire pixel information, a localized area of a face image can be affected by small lights, position, and any change in facial expression, resulting in affecting the recognition algorithm. This makes a face recognition method easily influenced by lighting, posture, and facial expression changes. On the other hand, since the model-based face recognition method can be configured in consideration of model, lighting, posture, facial expression change, it can reduce the influence of these factors at the time of recognition. Gabor feature vector (Coefficients obtained by the convolution of Gabor Wavelet Kernel about the face image feature points), a feature vector used when configuring the model, have been widely selected for facial recognition because it is somewhat robust for lighting, posture, facial expression changes [7]. EBGM (Elastic Bunch graph matching) is the leading face recognition method using Gabor feature vectors [8, 9]. Face recognition method based on EBGM performs the face recognition by finding feature points from the face, and by using Gabor coefficients obtained from these feature points.

Image-based face recognition techniques have the disadvantage of not being able to respond to a changing environment because it uses the entire image information. EBGM, the feature vector-based technique, which uses local-information, was developed in order to supplement this disadvantage. EBGM is done by obtaining the Gabor coefficients from the feature points, facial geometric information of the eyes, nose, mouth, and then by verifying the person who has the highest similarity when comparing the similarities between Gabor coefficients.

EBGM extracts only the local frequency characteristics of the image through Gabor wavelet function. It is then divided it into a real part and an imaginary part using the Gabor wavelet filter, and discretize each of them, and then create a Gabor wavelet mask. After that, coefficients are obtained by rising integral with the pixel values of the feature points from the face area image.

Gabor jets are the coefficients obtained from each feature point, and “Bunch” is a bunch of Gabor jets. The Bunch graph is the collection of a bunch of facial feature points. The model bunch graph is made by creating specific training image of M and manually putting dots on these bunch graphs.

A model bunch graph is used to obtain the feature points of the face. Feature point extraction used in RBGM is influenced largely by the number and type of models bunch graph. Therefore, the model bunch graph is needed to be created in consideration of a variety of gender, age, lighting, posture, and facial expressions.

2.3Verification of the identity of the face detected by comparing the feature points

The point matching method is the most widely used comparison method in the area of computer vision and pattern recognition. The 2 types of point matching are divided into rigid matching and non-rigid matching, depending on the strain of the target object extracted from the image sequence. Non-rigid matching is much more complicated, compared with rigid matching. In the case of non-rigid matching, it is difficult to compare because the relationship between each feature points has the severe degree of variation, depending on the strain of the target object. Facial features correspond to very severe non-rigid matching because the variation is severe depending on facial expressions and the angle. The effective non rigid matching algorithm is needed in order to solve them.

In this paper, for effective verification of identity, we performed the point matching using the Topology Preserving Relaxation Labeling (TPRL) algorithm, the modified version of Robust Point Matching-preserving Local Neighborhood Structure (RPM-LNS) algorithm, in comparison of feature points.

2.4Topology preserving relaxation labeling (TPRL)

In the method of RPM-LNS, the compatibility coefficient value was a binary value of zero or one, depending on whether a point and its neighboring point have corresponding points [10]. In TPRL, the correlation between point pairs is measured by the proposed compatibility function, which quantifies the amount of similarity and spatial smoothness between the point pairs in n-discrete values. This contextual information combined with a relaxation labeling process is used to search for a correspondence. Then the transformation is calculated by the thin plate spline (TPS) model [11]. These two processes are iterated until the optimal correspondence and transformation are found. The proposed relaxation labeling, with a new compatibility coefficient, preserves a topology of point set.

2.5Measurement of HRV

PPG measurement with a camera extracts a PPG signal p⁢(t) by analyzing the record of the person’s face towards the camera. The recorded video image is shown in the form of V⁢(x,y,t) for each frame in accordance with the strength of the signal. Each video frame saves the light reflected from the face as the pixel value on each of the x, y coordinates, and if the camera sensor takes measurement of many colors (e.g. red, green, blue), it can obtain a number of data value separated by color channels in a single frame (e.g. Vr⁢(x,y,t), Vg⁢(x,y,t), Vb⁢(x,y,t)).

The video image V⁢(x,y,t) is also consisted of two elements of the illumination intensity and the skin reflectivity.

The intensity of illumination I⁢(x,y,t) indicates all the lightings that illuminated on the face. When measuring PPG, certain intensity of illumination is needed for the area of PPG measurement. The reflectivity of the skin indicates the intensity of illumination reflected from the skin, and there are two types of reflection: the reflection from the surface of skin and the reflection from subcutaneous tissue.

The HRV measurement method using a camera in the detected face area is basically the same as the PPG measuring principle. PPG measurement is a method for estimating the heart rate by measuring a blood flow in the blood vessel by using the optical properties of biological tissues. The flow of blood between the diastolic and systolic heart gets slow when the blood flows into the peripheral vascular, causing the transparency changes in the blood vessels. PPG is basically sensing the transparency change in the blood vessel.

This form can be clearly measured in peripheral tissues in areas such as the face, fingers or earlobe, and the general PPG measurement is performed using a finger. If the same principle is applied to the face, the blood flow in the facial skin will also be changed depending on the heartbeat, and this can be measured and analyzed through the camera, and then the heartrate can be estimated.

Divided the blood flow evenly into the area of interest (ROIs) in the intensity of the extracted signal V⁢(x,y,t) and show it as R. Here, yi⁢(t) is the mean value of the pixel values of ROI Ri according to the time t, and i∈{1,2,…,n} is the index value of R.

In addition, if the intensity of illumination I⁢(x,y) of ROI Ri is constant and is shown as Ii, yi⁢(t) can be defined as follows.

In the formula above, Ii represents the intensity of illumination of ROI Ri, αi indicates the intensity of blood flow. In addition, bi is the reflectivity rate from the surface of facial skin and qi⁢(t) represents camera quantization noise. When the light of Ii is shone on ROI Ri, a large amount of reflection (bi) takes place at the surface of the skin. However, this value is a value that is not at all affected by the changes in the blood flow. Some of the light enters through the skin surface, and reflectivity depends on the changes in blood flow, represented by p⁢(t). Furthermore, αi seems to change depending on the absorption of the light and the wavelength of the irradiating light of Hemoglobin and Oxyhemoglobin. Thus, subcutaneous reflectivity can be calculated with two elements αi and p⁢(t) which contributes to the reflection of the light under the surface of skin. In this system, we used green channel which has the highest wavelength in the light absorption rate of two color factors, hemoglobin and oxyhemoglobin.

In order to estimate the exact heart rate condition, it is necessary to remove the various noises and only find a physiologically valid signal. Blind Source Separation (BSS) technique is one of the techniques performed in these operations. BSS is the method which separates and finds the valid signals on the assumption that the measured signals cannot be found instinctively and the valid signals mix as they are measured. Although there are several methods in performing BSS, in this paper, the blind source will be extracted using Independent Component Analysis (ICA).

2.6Part where the heart rate is measured

In the face, the area in which the capillaries are dense is the forehead and cheeks. Because of this, the body temperature of a person is usually measured in the forehead or cheeks. Similar to measuring the body temperature, when measuring PPG in the face, it is appropriate to measure the area which has a big change in hemoglobin concentration in accordance with the heart rate due to the dense capillaries. Thus, this system uses forehead and cheeks for measuring sites as candidate parts. But the cheeks were ultimately chosen as a measurement part because the forehead can be covered with hair depending on the person.

When the feature points of the face are extracted, the following placeholders are used: n⁢(t) for the nose, e⁢(t) for ears, m⁢(t) for the corners of mouth. The center of gravity (G⁢(x,y)) of the triangle is formed by these three points n⁢(t), e⁢(t), m⁢(t) and 10 × 10 pixels are extracted from this center point. The values from the 10 × 10 pixels are then taken to come up with the average.

2.7Blood pressure measurement

The cuff-based blood pressure measurement method is the most commonly used non-invasive method of measuring blood pressure. However, the disadvantage to this method is that the continuous blood pressure data for the blood pressure requires an interval of 1–2 minutes between the measurements in order to reduce the error. Because of this, this method can’t be used to measure the change in the blood pressure in a short period of time, resulting in not being able to do an immediate response to blood pressure changes. In order to resolve this problem, a method of measuring blood pressure have been developed to measure the blood pressure by using a time difference between the pulse rate on the fingertips and the heart [12].

In this paper, like the conventional method, the blood pressure was measured by using the pulse time difference between the heart and fingertips, and the pulse time difference between the heart and face.

For improved convenience, we developed a novel blood pressure measuring method without ECG patch. However without ECG patch, we can’t calculate pulse transit time. So in this method, we estimate ECG peak time for calculated pulse transit time.

In the formula above, th represents the pulse time of hand, tf is the pulse time of face and t𝐸𝐶𝐺 is R peak time of ECG. In addition, sh is the distance of heart to hand and sf is the distance of heart to face. k is coefficient about other factors that influence to pulse transit time.

2.8S⁢p⁢O2 measurement

Generally, the oxygen saturation rate S⁢p⁢O2 can be obtained by calculating the ratio of oxy-hemoglobin and hemoglobin.

H⁢b⁢O2 represents oxy-hemoglobin, and H⁢b represents normal hemoglobin that does not contain oxygen. H⁢b⁢O2 absorbs 880 nm wavelength and, H⁢b absorbs 765 nm. Using these principles, the 765 nm and 880 nm wavelengths were used as a light source system for emitting light alternately to measure the oxygen saturation from the image.

In the system, we made the light wavelengths of 765 nm and 880 nm to emit light alternately in a half sampling rate of the camera by positioning 6 × 6 Light Emitting Diodes (LED) with a wavelength of 765 nm and 880 nm alternatively. When you shoot an image matching the start of image and LED pulse, the image taken by 765 nm becomes an even-numbered image, and the image taken by 880 nm becomes the odd-numbered image. These images are separated into odd-numbered and even-numbered, and then the mean value is calculated from 40 × 40 pixels of the image. After this, the wavelength is passed through 0.5∼3 Hz of the Butterworth band-pass filter to the value of the G channel, and the value RO⁢S is obtained using the Eq. (6).

RV is the minimum point of 765 nm, and is the maximum point of 765 nm. RP is the minimum point of a 880 nm infrared ray, I⁢RV is the maximum point of a 880 nm infrared ray. The blood oxygen saturation will be shown in real-time by mapping RO⁢S calculated using the Eq. (5) on actual S⁢p⁢O2.