Reinforcement learning for an intelligent pediatrician robot

1.Introduction

Dealing with a child’s health is often not an easy task, because children often cannot express their symptoms very well. The doctor therefore needs to obtain most of the relevant information through data readings from a sensor device or by asking the child’s parents about the symptoms.

Yang et al. (2020) have proposed the creation of an integrated environment incorporating smart clothing, cloud databases and personal robots. It is claimed that the proposed system can act as a bridge between human and robot to express the patient’s emotional state, so the robot can help to decrease any stress or tension felt by the patient. The patient’s emotional state is captured through a biosensor attached to their body, and transmitted through a cloud server (Yang et al., 2020).

Artificial Intelligence has made it possible for robots to evolve and be capable of helping people in education, medical diagnosis, and even providing medical treatment for specific diseases and mental health issue (Chen et al., 2018b; 2018a). Some researchers have focused on body sensor networks made by combining multiple sensors for handling smart medical services. Such networks are needed for multi-platform communication between a robot and various of the sensors (Lin et al., 2020). The desire of the medical community for using body sensor networks has increased tremendously in recent years due to the use of the internet to communicate between a wearable sensor and a cloud server. Body sensor networks are capable of capturing physiological and psychological data, and even some activities of the patient prior to conducting an analysis based on that data (Zhang et al., 2018; Fortino et al., 2012).

This paper focuses on a preliminary investigation of the possibility of using a robot as a pediatrician, based on the capability of a reinforcement learning algorithm that can adapt a robot for use with children. Section 2 presents a brief literature review of related work. Section 3 discusses the materials used for our investigation and the methodology of our research. Section 4 describes the initial implementation of the robot, while Section 5 presents our overall research conclusions.

2.Related works

The use of robots as assistants for our daily lives has grown for the past few years especially for the elderly, leading some researchers to analyze the ethical, health policy, and social impact of robot use in society, and to the conclusion that robots should not replace humans, but should only be employed as support for accomplishing specific tasks (Lehoux and Grimards, 2018). And robots become more challenging when used as supportive tools for the elderly because their control and manipulation can be quite complicated for an older adult. Assistant robots might be used for help with physical tasks, with observation, for security alerts, or as companions (Shishehgar et al., 2017; Smarr et al., 2014; Wang et al., 2017). Fridin (2014) has proposed an assistive robot (KindSAR) for kindergarten children, aimed at assisting preschool children with learning and play, through the social interaction between robot the and the child (Fridin, 2014).

Social robots are becoming attractive to many of us. There are attempts to involve robots in more intense interactions with humans, more collaboration on certain specific goals (Picarra and Giger, 2018), tasks such as preparing medication or serving food, teaching children foreign languages, and assisting the elderly with their daily activities (Simon, 2015; McDaniel, 2010; Wada et al., 2005). Robots are also used as walking assistants for children with multiple disabilities. For example, a humanoid robot that can demonstrates a specific way of walking slowly, suitable for the child (Wang et al., 2016).

Based on such examples we can conclude that robots are being used quite widely as support devices and even as assistants. Due to shortages of medical staff in various countries and regions, robots have become essential in providing assistance to satisfy the needs of the medical profession. The pharmaceutical giant Pfizer has made a breakthrough with a one-year pilot study using an intelligent robot, Mabu, as an assistant that will help with in-patient health and the prescription of medication.

In providing a framework for the robot as a pediatrician we consider how the robot can assist in the diagnosis of the child’s disease – how the robot can persuade children to undergo a particular test, and how this robot can communicate with children in a friendly way. Most children are quite shy or even scared when meeting a pediatrician, but children will not be so shy or afraid of a humanoid robot. That is why our proposed framework calls for the robot to send a child’s data to the server via the robot.

The Children’s Hospital Association, which has more than 200 hospitals in the USA, Canada, Australia, and other countries, reports vacancy rates as high as 46.9% for pediatrician positions in its hospitals, as illustrated in Fig. 1. (Knapp and Major, 2018). This statistic is the highest vacancy percentage (along with child psychiatry), in the Children’s Hospital Association.

Fig. 2 shows the highest-ranking shortage that is serious enough to affect a hospital’s ability to deliver adequate care is held by Developmental Pediatrics departments, with 11.8% (Knapp and Major, 2018).

Fig. 1.

Hospital vacancy, image courtesy of Children’s Hospital Association (Knapp and Major, 2018).

Fig. 2.

Ranking of medical staff shortages, image courtesy of Children’s Hospital Association (Knapp and Major, 2018).

3.Material and method

The material that we used for our preliminary experiments consists of the NAO humanoid robot and Internet-of-Things devices such as pulse sensors and the Arduino UNO R3 (see Figs 3 and 4).

Our reinforcement learning system is designed for an evolutionary agent that can adapt to its environment. The agent will continue its activities, receive its rewards, and adjust its behavior. The general model of reinforcement learning for the pediatric robots is illustrated in Fig. 5.

Fig. 3.

NAO actuator and sensor image courtesy Aldebaran (2019).

Fig. 4.

Temperature sensor and Arduino board.

Figure 5 shows the basic tasks of the proposed framework – the paediatric robot with reinforcement learning. Initially, the starting position of NAO might be seated as shown in Fig. 6.

Fig. 5.

General framework of reinforcement learning for pediatric robot.

The robot will carry out specific activities such as checking the child’s health by measuring the heart rate or their body temperature. Then the robot might perform particular tasks that require it to move. When the robot performs well, or at least acceptably, it receives a reward such as applause or an appropriate rating from those observing its activities.

Fig. 6.

Initial mode of NAO Robot in seated position.

The paediatric robot continually feeds the learning system them with new data as the surrounding circumstances change. So, it will make the robot dynamically adapt to the situation and react according to the latest circumstances.

The proposed framework is designed based on a reinforcement learning algorithm. it is initialized with an input and then continued with rewards whenever the robot perform an action. The action will be improved from time to time according to environment. In other words, it has capability to adapt or evolve toward particular condition.

4.Result and discussion

The above discussion indicates that reinforcement learning has great potential to teach a robot to become a dependable pediatric robot that can get along well with children. Naturally the robot itself naturally attracts children and makes them accessible to the robot and comfortable with their interaction. Children tend to put aside their fear of being monitored when a robot is involved in the process, for example when they have their body temperature taken or their heart beat measured.

In the future a pediatric humanoid robot such as NAO should able to do more than checking the child physically. It should also be able to provide pediatric nursing. For example, by providing therapy to the child’s neuro system.

A more complex movement for motoric training is illustrated in Fig. 7. The Robot asks the child to listen for specific instructions and perform a simple push-up action, whereupon NAO will applaud the child for following its movements.

NAO can employ an Arduino sensor and board, as shown in Fig. 8, to enable NAO to monitor a child’s heartbeat. The beats-per-minute data is transferred to the cloud server for recall by NAO as required.

Fig. 7.

Motoric training by NAO.

Fig. 8.

Heartbeat measurement.

In order to verify our proposed framework, we decided to focus on the children’s motoric function, which we analyze based on a data set derived from children with various physical and motor difficulties (Zarichi et al., 2018) We focused on the class 6 problem from that data set, which consists of several difficulties faced by the children in that study (Zarchi et al., 2018):

The simulation process on this dataset demonstrates that a neural network is superior (giving 83% accuracy) compared to both logistic regression and naïve Bayes, refer to Fig. 10 . As mentioned earlier, we used a reinforcement learning algorithm for our robot to increase its ability to evolve and adapt to change. During the simulation it was shown that a repeated learning process such as a neural network has supported the proposed algorithm to improve the learning process and achieve better results.

Fig. 9.

Dataset for children with physical and motor disabilities.

Figure 11 shows further evaluation analysis with a confusion matrix comparison among three machine learning algorithms. First, the neural network has acquired 96.6% prediction accuracy, it has same achievement as logistic regression. Unsurprisingly, the naïve Bayes was only able to achieve 62.1%. this comparison shows that the proposed algorithm has worked as expected.

Fig. 10.

Evaluation result for different machine learning algorithm.

Figures 12 and 13 display our evaluation results for the simulation. The neural network is indicated by a green line, logistic regression is purple, and naïve Bayes is yellow. Initially, the True Positive rate (TP) for the neural network increases until the Positive prediction (P rate) reaches 0.3, and TP reaches its peak when the P rate reaches 0.4, when TP has the same value as with logistic regression.

Fig. 11.

Confusion matrix of the three-machine learning algorithm.

Fig. 12.

Lift curve.

Fig. 13.

ROC analysis.

Figure 13 display the Receiver Operating Characteristic (ROC) analysis of the simulation, where the neural network has shown high sensitivity compared to Naïve Bayes and logistic regression since the beginning of the false positive rate (FP), while logistic regression begins to change at an FP rate of 0.02.

5.Conclusion

Our pediatrician robot is an example of the plan to assist human pediatricians in helping children with the monitoring of their health, and with other forms of pediatric assistance. Much work remains to be done, but the pediatrician robot has slowly but surely been realized.

This paper describes work which is still in its initial stages, introducing reinforcement learning as our chosen learning algorithm for the NAO robot. The combination of a humanoid robot and Internet-of-Things sensors can help the pediatrician robot to become a reality. A humanoid robot can measure the child’s physiological data. It can conduct further analysis and carry out further tasks such as treatment recommendations or writing initial drug prescriptions. This is when the reinforcement algorithm comes in handy with its capability to adapt to environmental conditions. The preliminary implementation of this research focused on the motoric functions of children, who happily follow the instruction of the humanoid robot. Furthermore, our simulation showed that machine learning algorithm such as neural networks could support reinforcement learning to achieve up to 96.6% recognition of a child’s specific disability problem. We anticipate the need for much future work, both in terms of sensor integration with a humanoid robot and with intense robot training with reinforcement learning algorithms in order to train the robot to react and adapt to any situation.