Intelligent quality control of traditional chinese medical tongue diagnosis images based on deep learning

1.Introduction

The four diagnostic methods in Traditional Chinese Medicine (TCM) are divided into “inspection, auscultation and olfaction, inquiry, pulse-taking and palpation” [1]. Among them, “Inspection” is listed in the first of the four diagnoses of TCM. It is crucial for TCM doctors to preliminarily know the patient’s physiological state, diagnose the patient’s disease types and pathological changes. This is because the functions of the internal organs often have their external manifestations.

Tongue and facial examination are essential parts of “Inspection” [2]. In TCM theory, doctors must observe the shape, color, and texture of the patient’s tongue. High-quality tongue photography requires patients to open their mouth in a “non-smiling state”, extend the tongue naturally and expose the root of the tongue. In addition, asking patients to open their eyes can provide more information for follow-up facial examination. However, traditional facial and tongue examinations are directly observed by human eyes in clinical practice. With the popularity of tongue diagnosis instruments and the emergence of mobile applications related to TCM, the analysis of tongue and facial images in TCM has moved towards digitization and quantification. Therefore, how to obtain a qualified image so that the computer can conduct subsequent quantitative analysis and avoid producing incorrect results and ambiguities becomes particularly important.

The acquisition of tongue image is the most basic and critical part in the objectification of TCM tongue diagnosis. The acquisition of high-quality tongue image is conducive to the subsequent detailed analysis of tongue characteristics and high-accuracy pathologic research. In the past 10 years, many tongue image acquisition systems have emerged in the market [3, 4]. However, these systems have different light sources and color temperatures, resulting in considerable differences in the obtained tongue images. After the tongue image is acquired, some image processing algorithms need to be used, among which the most essential work is tongue detection and segmentation. These methods can be roughly divided into the following types.

The first type is based on the classical mathematical model, such as threshold method [5], since the color of the tongue is usually red. For instance, Gu et al. [6] utilized an automatic tongue segmentation algorithm based on the threshold, whose experiment results show that the segmented area is complete and close to the actual target. Fachrurrozi et al. [7] used hybrid multilevel thresholding and harmony search algorithm to segment the tongue. Snakes or other active contour models [8] can dynamically change the deformation model to find the boundary of the region of interest, to achieve segmentation based on prior knowledge. For example, Sapaico et al. [9] combined the mouth appearance model with edge detection algorithm to fulfill tongue detection. However, the threshold segmentation method depends on the quality of the image, while the statistical model relies on the initial curve.

The second type is based on the traditional machine learning. In the early 21st century, machine learning has been developed rapidly. Machine learning has also made remarkable achievements in tongue image detection and segmentation. Zhou et al. [10] used a reconstruction enhanced probabilistic model for semi-supervised tongue segmentation. Hu et al. [11] put forward an unsupervised tongue segmentation method validated in comparisons with the snake method. Liu et al. [12] proposed a patch-driven segmentation method based on sparse representation. Sapaico et al. [13] utilized a 3-layer cascade of SVM learning classifiers to detect the tongue. Li et al. [14] used histogram projection and learning-based digital matting to extract a tongue from images. Liu et al. [15] put forward a tongue recognition and localization method based on the LBP feature and cascade classifier. However, traditional machine learning algorithms require manual design of the tongue’s features, with limited feature expression capabilities.

The third type is based on the deep learning. In contrast to traditional machine learning algorithms, deep learning enables the automatic extraction of image features, eliminating the requirement for manual design. For instance, Welikala et al. [16] assessed two deep learning structures based computer vision approaches for the automated detection and classification of oral lesions. Shamim et al. [17] have applied and evaluated the efficacy of six deep convolutional neural networks models using transfer learning for identifying pre-cancerous tongue lesions directly. A Multi-Task Joint learning method for segmenting and classifying tongue images is proposed by Qiang Xu et al. [18]. Tang’s [19] and Kong’s [20] team both used CNN to complete tongue detection. Cai et al. [21] applied a robust interclass and intraclass loss function to improve the performance of deep learning frameworks. Li et al. [22] used the U-Net model to achieve tongue image segmentation, which obtained 98.26% accuracy. Zhou et al. [23] added a morphological layer to the last layer of traditional U-Net, accomplishing segmentation of small-scale tongue image. Huang et al. [24] proposed a Tongue U-Net (TU-Net), which combined the U-Net structure with Squeeze-and-Excitation block, Dense Atrous Convolution block and Residual Multi-kernel Pooling block to segment the tongue region. More recently, Tang et al. [25] and Jiang et al. [26] both utilized deep learning technology to realize tongue image detection and quality control. Deep learning has achieved some good results in the specific field of tongue image analysis, but its clinical application scenarios in traditional Chinese medicine still need to be further expanded.

In summary, there are still many shortcomings in the design of the tongue image analysis system. First, there is no universal standard for tongue image. Different acquisition devices have different light source color temperatures and different display indices. The imaging camera models are changeable. Moreover, the acquisition environment can be open, semi-open, or closed. Second, many images are unqualified to even be of no clinical value in the practical application. As depicted in Fig. 1, the correct and qualified image should reveal the tip, middle, and root of the tongue, while images where the tongue is not extended or straightened are considered unqualified. However, there are few acquisition devices to judge the eligibility of the image. Third, the common tongue diagnosis instrument only considers the tongue and does not collect a complete facial image. In the process of TCM inspection, the collection of facial features is vital. Also, eye-opening or closing will significantly affect the results of the facial diagnosis.

Figure 1.

Common tongue images, where (a) and (b) are qualified tongue images, and the tongues in (c) and (d) are not extended, rendering them unqualified.

The basis of computerized tongue diagnosis is to collect a complete tongue image. This paper proposes an intelligent quality control algorithm for TCM tongue diagnosis images based on deep learning to judge the qualification of images. The method uses the ResNet framework to evaluate the qualification of collected image data to improve the quality of data sets and guarantee subsequent tongue image detection accuracy. The contributions of this paper mainly include: Firstly, it presents a fast and precise quality control method for traditional Chinese medicine images based on deep learning, which combines artificial intelligence with traditional Chinese medicine. Secondly, it integrates the joint quality control of facial and tongue images, and can indicate the reasons for unqualified images. In this paper, we are going to discuss our problem as follows: Materials and Methods which propose a fast and accurate quality control method for TCM images based on Resnet34. Results which show the effectiveness of our method; Discussion which analyzes the advantages and limitations of this method, and puts forward future research directions.

2.Method

2.1Algorithm description

The essence of judging whether an image is qualified is a multi-classification problem. This paper proposes an image quality control algorithm based on deep learning to verify the eligibility of TCM tongue diagnosis images. The algorithm workflow is shown in Fig. 2.

Figure 2.

The workflow of our method. The workflow comprises two stages: the training stage depicted above utilizes a Resnet34-based transfer learning model with training data including five types of images; the testing stage uses the well trained model to identify new images and indicates the reasons for unqualified images.

The algorithm includes the training stage and testing stage. In the training stage, we first collected enough images and divided them into five states according to whether it was a human face, whether the tongue was straightened and whether the eyes were open. We do these two steps to form a training data set. Secondly, perform image preprocessing on all training images. Then build a convolutional neural network model and use the normalized training image as input to train the network model. In the testing stage, firstly take the patient’s images and normalize them. Then, input the test images into the trained convolutional neural network model to achieve automatic image classification. Our method uses the neural network model to analyze and judge the eligibility of the image. It improves the quality of the TCM tongue diagnosis image and enhances the accuracy of tongue recognition by digging into the high-level features of the tongue.

2.3Resnet residual network

The depth of deep learning is critical to the performance of the model. When the number of network layers increases, the network can extract more complex feature patterns. But if the network depth is too deep, it will lead to degradation problems. ResNet significantly reduces the attenuation of this gradient correlation, making it possible for the network to have hundreds or even thousands of layers.

The residual network is composed of a stack of residual units. The residual unit is shown in Fig. 3a.

The residual unit can be expressed as:

(2)

yl=xl+G⁢(xl,Wl)

(3)

xl+1=g⁢(yl)

Figure 3.

(a) Residual units. The core of residual units is to introduce a skip connection in each layer of the network, passing the output of the previous layer directly to the next and adding it. This helps retain information and alleviate the gradient disappearance problem. (b) The structure of Resnet34. Resnet34 consists of 34 convolutional layers and adds skip connections in the network, enabling information to be transferred more efficiently between layers.

Where xl and xl+1 represent the input and output of the l-th residual unit, and each residual unit contains a multi-layer structure. G is the residual function, meaning the learned residual and g is the ReLU activation function. Wl is the weight of residual unit. Based on the above formula, the L-layer features can be obtained by learning at the l-layer (L > l):

(4)

xL=xl+∑i=lL-1G⁢(xi,Wi)

Using the chain rule, the gradient of back propagation can be obtained:

(5)

∂⁡𝑙𝑜𝑠𝑠∂⁡xl=∂⁡𝑙𝑜𝑠𝑠∂⁡xL⋅∂⁡xL∂⁡xl=∂⁡𝑙𝑜𝑠𝑠∂⁡xL⋅(1+∂∂⁡xl⁢∑i=1L-1G⁢(xi,Wi))

Where loss represents the loss function. ∂⁡𝑙𝑜𝑠𝑠∂⁡xL can ensure that the information can be transmitted to any shallow unit during back propagation. ∂∂⁡xl⁢∑i=1L-1G⁢(xi,Wi) cannot always be -1, so the problem of gradient disappearance can be avoided in the residual network.

The path that directly maps the upper layer to the lower layer in the residual network is called a shortcut. Due to the existence of the shortcut, the upper layer information can be passed to the next layer without loss. It will not affect the entire training. The residual network can be regarded as another network composed of adaptive algorithms. No matter how many layers are stacked, the actual layer with parameters is determined by the training itself. Many residual units and a fully connected classification layer can form a ResNet34 network. The specific structure is shown in Fig. 3b.

3.Computer simulation and results

3.4Algorithm performance

We reproduced several classic deep learning classification models, including Alexnet, VGG16, and GoogLeNet [27]. All models used the same hyperparameters. The experimental results are shown in Table 1. For the test dataset, the classification accuracy of our algorithm can reach 97.06%. Moreover, the classification speed of each picture is in milliseconds.

Table 1

Classification results in test dataset compare with other models

Model	Accuracy	mP	mRecall	mF1
Alexnet	94.11%	91.81%	93.24%	92.52%
VGG16	94.11%	93.33%	93.57%	93.45%
GoogLeNet	95.59%	95.14%	93.95%	94.54%
Ours	97.06%	96.64%	95.95%	96.29%

Figure 4.

Loss curve and confusion matrix of the classification results. (a) represents the loss curve during training, and it can be found that during the training process, the convergence of the loss function is very fast. (b) represents confusion matrix of testing data. The row represents the predicted results, and the column represents the ground truth.

From Fig. 4, we can see that our trained model achieved highly promising classification performance. In the category, the first two letters represent the state of the eyes, where “EY” stands for qualified eyes and “EN” stands for unqualified eyes. The last two letters indicate the state of the tongue, where “TY” represents a qualified tongue and “TN” represents an unqualified tongue. Only ‘ENTN’ and ‘EYTN’ produced one wrong classification result among the five classes.

Figure 5.

Visualization of learned representations using t-SNE. (a) represents the train set (b) represents the test set. Different colored dots indicate the distribution of features in various categories via deep learning. Features within the same category are relatively proximate, while those in different categories have a certain distance.

To have an intuitive understanding of the classification performance of our models, we visualize the learned features on the train and test set in Fig. 5, respectively. We map the high-level features to 2D space by applying t-SNE [28]. The t-SNE technology can reduce the dimensions of data by constructing the probability distribution between samples, so as to better understand the structure and relationship of the data. From Fig. 5, we can clearly see that features from different categories are separated well, reflecting our proposed method’s strong discriminative power of the learned representation. Additionally, we can learn that the classes ‘EYTN’ and ‘Nature’ are somewhat inseparable, and they are still separated from their original class, which indicates some noise data. These noise data can encourage the model to be more robust.

4.Conclusion

With the progress of image processing, researchers use computer vision to create a new approach for objectifying and standardizing TCM tongue and face examination. However, current research has the following shortcomings: non-standardized image collection lacks qualification testing and standardization, making these images less suitable for tongue detection.

To achieve the quantification of data and optimize the process, this paper proposes an intelligent quality control algorithm for Chinese tongue diagnosis images based on ResNet34 and transfer learning. First, the tongue image is preprocessed, the data is uniformly scaled to 224 × 224 size, and regularization is used for image preprocessing. Secondly, using ResNet34 as the backbone of the neural network and adding a linear layer at the end for classification, the images are divided into five categories: non-face images, tongue not straightened, eyes closed, tongue not straightened and eyes closed, and qualified.

The method proposed in this paper has the following advantages: (1) The use of deep learning methods for image detection has fast speed, better robustness, and higher accuracy. (2) Compared with previous studies, the image standardization research ensures subsequent tongue image segmentation and feature extraction. Due to the limitation of one’s ability and the influence of external objective factors, the method proposed in this paper still has some shortcomings. For instance, incorrect judgments by the algorithm still occur. Two examples of errors in the experiment are as follows: a child’s image with a straight tongue is identified as having a non-straight tongue, and an image of an animal is recognized as a human face. Furthermore, when collecting images, subjects are all required to remove their glasses. However, in real life, it is highly probable that people will be photographed wearing glasses. In the following study, we will continuously improve the analysis accuracy and take more factors, such as adopting more robust classification models, or introducing attention mechanisms, into account to achieve higher efficiency and precision.