Sensors, vision and networks: From video surveillance to activity recognition and health monitoring

2.1.History of hardware progresses in IVS

As said above, this section will briefly browse the temporal changes along the last years in terms of hardware progresses in IVS systems.

At the beginning, also due to cost and availability issues, most of the works in IVS tackled the single fixed camera scenario. The video feed provided by a single fixed camera was processed to automatically analyze the scene [19,22,37,55,118,134]. However, the single fixed viewpoint provided by a single camera has some limitations, in terms of both the amount of area covered and the robustness to occluding objects. The former limitation does not allow the system to monitor large areas (usual in security applications) and prevents the detection and tracking of objects when they move away from the field of view of the camera. The latter limitation, instead, makes the system unreliable when the targets are (partially) occluded by other moving/non-moving objects.

In order to solve these limitations, around 2002 some works on IVS started looking at different types of cameras. For instance, PTZ (Pan-Tilt-Zoom) or active cameras have been used to dynamically change the viewpoint of the single fixed camera in order to get rid of occlusions and to expand the monitored area [23]. One limitation of PTZ cameras is that a single field of view is available at a given time, and proper scheduling algorithms need to be designed to have a proper (and continuous) observations of the targets [39]. One possible alternative consists in using special cameras, such as the omnidirectional cameras (see, for instance, [87]) which provide a 360-degrees simultaneous view of the scene thanks to an hemispherical mirror on the top of the camera lens.

Despite the advantages of using cameras with variable (PTZ) or large (omnidirectional) fields of view, they still have limited applicability on large-area surveillance and are still prone to the occlusion problem, due to their fixed optical center. Therefore, from the very beginning, there has been an ever-increasing interest in using multiple cameras (either fixed or active, or both), composing, as a matter of fact, a network of (hybrid) sensors. Although some earlier works (such as [71,78,117]) already considered the use of multiple cameras, the milestone for a more-or-less concrete system with multiple cameras can be considered the work of Khan and Shah in 2003 [68]. This work can be considered among the first ones to address the problem of fusing multiple camera tracking outputs to construct a multi-camera (multi-object) tracking system. In fact, it proposed an algorithm to the consistent labeling problem, i.e. assigning the same (consistent) label (ID) to an object detected in different cameras. This labeling is the initial step to obtain a long-term multi-camera tracking of objects within a large area, followed by analysis of the object trajectories and thus of its behavior in the scene. Several approaches have been proposed for multiple camera tracking, classified depending on whether the fields of view of the cameras are partially overlapped or completely disjoint. In the former case, geometric as well as appearance-based features can be used to solve the data association task between the detections in two views, while in the latter only pure appearance-based features can be used [25], together with some reasoning about the travelling time between adjacent cameras.

There is still a very active research community working in this field, with an increasing emphasis on the size of the network (i.e., the number of cameras), the variety of sensors (hybrid systems), the complexity of the scene (especially, with non-overlapping fields of view), and the final application (see next section with regards to people re-identification topic). Interested readers may refer to the 2009 book [2] or to the 2013 review reported in [131].

While the research on multi-camera IVS was progressing, another active area of research has come up around 2004 [20]. In fact, thanks to the advances in electronics, the so-called smart cameras have started to be affordable at the beginning of 21th century. Smart cameras are not only acquisition devices, but also include processing, storage and communication modules. These modules allow the IVS system to pre-process the acquired video feeds on board and transmit only relevant information to the server(s) through the communication network. Besides the obvious relief of the communication and processing load, smart cameras are also more privacy respectful, by retaining and transmitting only data which do not violate the privacy laws.

These two last research areas (multi-camera systems and smart cameras) have been often fused together by building hybrid IVS systems that exploit the advantages of smart cameras within a distributed networks of sensors (cameras) [57].

Fig. 1.

(a) Approximated timeline of hardware progresses for IVS. (b) Approximated timeline of algorithm progresses for IVS.

One further step in the hardware progresses for IVS relates to the general concept of mobile vision. This term was first introduced in [59] and refers to the development of computer vision and image processing techniques for camera-equipped consumer devices, such as the latest generation smartphones and tablets. Mobile vision shares and builds upon prior work on active vision, which deals with computer vision algorithms for processing imagery captured by non-stationary cameras. Similar to cameras mounted on a robot or a vehicle, camera-equipped consumer devices that are carried by humans are non-stationary. Furthermore, it is oftentimes more difficult to model the motion of a camera embedded in a hand-held consumer device than that of a camera mounted on a robot. Mobile vision also poses a whole set of unique challenges stemming from the fact that these devices have limited computational, memory and bandwidth resources [103]. There are several reasons why mobile vision is an emerging field of research. First of all, the latest generation of consumer devices – smart phones and tablets – have the computational resources needed to support “sophisticated” computer vision algorithms. Many of these devices also boast multiple sensors, including a number of non-imaging sensors, such as compass, accelerometer, and gyroscope. Readings from these sensors can be incorporated into the image processing pipeline to support a variety of applications, e.g., augmented reality, geo-localization, etc. Furthermore, advances in communication technologies have resulted in a new generation of high-speed, low-latency wireless communication protocols. This suggests that it is now possible to leverage cloud-based computational resources and lower the onboard computational burden on consumer devices. Even more importantly, the last 30+ years of research on computer vision algorithms has produced highly-accurate and efficient algorithms that can now be deployed on consumer devices. All these factors have contributed to the development of a multitude of mobile vision projects and products, such as visual object (landmarks, logos, goods) recognition, augmented reality, gesture recognition, road sign recognition, lane detection, automatic cruise control, collision avoidance, etc. [103].

One last advance in terms of hardware used in IVS has been introduced around 2012. In fact, the advances in consumer electronics has made available in the market the so-called egocentric cameras [66]. At the end, these are normal wearable cameras that provide a “first-person” perspective of the scene. A part from the technical challenges (common to the above-mentioned ones for mobile or robot vision), egocentric vision has also opened a brand new set of applications related, among the others, to IVS: face recognition, just to mention the most common, has been brought to a new level thanks to egocentric vision. Other applications are related to object recognition, as well as to personal storytelling from visual lifelogging [18].

Figure 1(a) summarizes the approximated timeline of the major steps in hardware progresses for IVS. It is quite evident that there has been a gap between 2004 and 2012 in the hardware progresses: this can be explained by, on the one hand, the focus of the scientific community on the algorithms (see next section) and, on the other hand, by the time needed to develop affordable and powerful mobile devices and egocentric cameras.

2.2.History of algorithm progresses in IVS

This section will briefly outline the progresses in the IVS field in terms of algorithms. As a premise, it is quite straightforward to notice that these progresses are tightly related to those in the hardware used (described in the previous section).

As in the case of other fields of computer vision (and computer science in general), the algorithm progresses have followed a low-to-high-level path. As a consequence, the first set of papers had “object/target detection” as primary goal. Since, at the beginning, single fixed cameras were used, the first efforts can be classified as the development of effective (and efficient) background suppression techniques. Background suppression refers to an algorithm (or set of algorithms) aiming at separating (at pixel- or object-level) the stationary parts of the scene (i.e., the background) from those which are moving (i.e., the foreground). A seminal work on this topic is dated back in 1999 [124]. Despite its age, this topic is still heavily researched in the literature, especially considering more complex scenes with slowly-changing backgrounds, distractors (e.g., waving trees, which are moving but should be classified as background) and moving cameras (where moving parts should be distinguished between actual background and moving objects). Some additional references (among the numerous existing in the literature) on this topic are [34,42,45,98,116].

Once the (moving) objects in the scene have been detected, the next step typically involves tracking objects, i.e., consistently associating the same identifier to an object, along time. There have been several papers addressing the object tracking problem. Although some earlier works are dated back to 1999 [24], a milestone for this topic can be set to 2003 when the work by Comaniciu et al. [38] has been published. Object tracking algorithms are often classified based on how objects are represented and on the data association phase [137]. Point-based trackers employ either a deterministic or probabilistic (based on Kalman filter or JPDAF – Joint Probability Data Association Filter) approach to establish correspondence between keypoints on the objects in successive frames. Kernel-based trackers, instead, use some more-or-less complex kernel and a classifier for data association. Finally, some trackers exploit the information about the silhouette of the objects to find correspondences. Interested readers may also refer to the experimental survey reported in [115].

As a natural extension, once the objects are tracked in each single camera of a network of multiple cameras, there has been a significant number of papers addressing the multicamera object tracking problem. The extension of object tracking to multiple cameras requires to take into account, when deciding the object representation and the data association strategy, the differences in viewpoint, resolution, illumination conditions and colors of the different cameras. Also in this case, it is rather subjective to set a starting date for this topic, but with the work of 2006 by Kim and Davis [69] (and others that appeared in that period of time), the proposed solutions can be considered mature enough. By considering multiple cameras, also issues related to their coordination/cooperation, as well as to the communication between them, should be taken into consideration [121]. Finally, this is still a very active area of research [108,122].

Most of the existing multi-object (multi-camera) tracking algorithms are capable of tracking objects in a low-medium-density situation. Therefore, since approximately 2008 [139], several researchers have studied specific solutions to the crowd analysis problem. This refers to situations where the density of people/targets is high or very high. The proposed solutions are mainly based on flow dynamics approaches [5] or on social force models [86].

Tracking objects on a large surveilled area is per se useful, but it is typically followed by algorithms that exploit the temporal information about the objects (appearance, position, speed, etc.) for higher-level tasks. Among them, in the recent past, object trajectories (as sequence of positions in time) have been analyzed as a mean for extracting information about the object behavior. Trajectory analysis has been a flourishing area of research, starting from the seminal work of 2005 [81]. The approaches to trajectory analysis are based on a specific representation of the trajectories followed by a clustering algorithm. Each obtained cluster of trajectories represents a common behavior, whereas small clusters identify anomalies (and, thus, potentially suspicious behaviors) [26]. Interested readers can refer to the survey in [88].

There are also several research topics under the IVS umbrella aiming at exploiting low- and mid-level algorithms (object detection and tracking, trajectory analysis, etc.) to design high-level algorithms, such as posture analysis [41,89], fall detection [91], detection of abandoned packages [16], action/activity recognition [30,101] and others. At even higher level, some papers studied how to model and classify interactions both between targets (mainly people) and with the scene, for instance for assisted living applications [29]. In particular, group detection and recognition of social interactions have been hot topics in recent years [8,40]. The main approach for these tasks is based on understanding the spatio-temporal personal space of the targets and using social force models (SFMs) to understand the interactions among them.

Finally, a special attention must be given to person re-identification topic, since it is still a very hot topic in the scientific community [47,129,142]. On the one hand, this can be treated as a special case of multi-camera tracking (i.e., the goal is to assign a consistent label to a person when it disappears in one camera and re-appears in another), but, on the other hand, it is also a borderline topic with the general image retrieval topic, and most of the techniques used are borrowed by that topic.

Figure 1(b) summarizes the approximated timeline of the major steps in algorithm progresses for IVS, and it can be noticed that it well integrates with the progresses in hardware.

2.3.Future trends in IVS

Most of the progresses listed in the previous sections are still more-or-less active topics in IVS. However, generally speaking and without claim to impose or address the future directions of research in IVS, there are some topics which might be considered pivotal for IVS in the near future:

4.1.Single accelerometer approach

Understanding the physical context such as body position, posture, gait, vital signs and physical activities of a person can offer great amount of information for context-aware applications such as smart home, elderly care, rehabilitation, and sports science. In this paper, we focus on the identification of the so-called Activities of Daily Living (ADL). Enabling or assisting elderlies and disabled patients to be able to perform ADL is critical in maintaining their quality of life. Majority of the ADL can be classified into three categories namely static, dynamic, and transitional activities. Static activities consist of activities such as standing, sitting, and lying, whereas dynamic activities consist of activities such as walking, running, jumping. In addition, there are also transitional activities that sit in between static and dynamic activities, such as sit-to-stand and lie-to-sit, which are of critical importance in inferring the correct physical context.

Naturally, most of these ADL are determined by human locomotion and therefore inertial sensors, or more specifically the accelerometer, are the most popular choice. Table 2 covers some of the prominent research works over the last two decades that use only a single accelerometer for recognizing human ADL. The main advantage of single accelerometer systems is their simplicity and unobtrusive nature that cause less impediment to users’ daily activities. Since there is only one accelerometer, sensor placement is important in achieving a more accurate recognition. While different sensor locations may lead to varying recognition accuracies of different activities, it can be observed that the most popular sensor placement location is around the pelvic region. The locomotion of the pelvic region is relatively steady, while still capable of providing discriminating features to separate various ADL.

HAR systems are typically considered as a classification problem where various types of feature selection, feature extraction, and machine learning based classification approaches are developed. While most of the developed techniques are able to achieve reasonably high accuracies, such results must be interpreted with care. Due to the computation power and energy constraints, most of the existing works are still considering the deployed WSNs as data acquisition tools and analyses are very often offline with emulated activities. Therefore, the real-time performance with real life data can be a key challenge for many existing works.

4.2.Multiple sensor fusion approach

While single-accelerometer-based systems have been proved to be very effective and less obtrusive to users’ daily lives, multiple sensor fusion approaches are still heavily investigated for various reasons (see Table 3 for some relevant examples). In the early days, a common alternative approach is to use multiple one-dimensional or two-dimensional accelerometers for HAR, as opposed to a single triaxial accelerometer. The construction of such systems are usually more difficult even with wireless communication capability.

Another typical fusion is to integrate accelerometer and gyroscope into the so-called Inertial Measurement Unit (IMU). Gyroscope provides more accurate detection on body orientation and posture which complements well with the accelerometer. However, as it can be observed in Table 3, fusing multiple sensor data does not necessary offer better accuracy. How to better fuse heterogeneous sensor data to achieve more accurate classification remains an important research topic.

Last but not least, accelerometers or IMUs are also commonly fused with other ambient sensors (e.g. RFID, GPS) to better infer more complex human activities/behaviours, or vital sign monitoring sensors (e.g. ECG, heart rate) to detect specific types of activities such as fall detection and gait analysis.

4.3.Fine-grained behaviour recognition

As mentioned previously, many different machine learning techniques had been applied for classifying simple ADL. It is worth noting that most of those approaches for classifying human physical activities are data driven (or learning-based). Data driven approaches are preferred for their abilities of handling noise and uncertainties, which are of critical importance when dealing with raw sensor data. On the other hand, data driven approaches also often require large amount of training/testing data in order to build a reasonably accurate classification model. The necessary preparation work such as collecting and labelling data can be very time consuming, which limits the applicability of these approaches [32,136]. In addition, recognition of simple physical activities may not provide sufficient context information to differentiate complex behaviours. For example, knowing the physical activity of standing may not be sufficient to differentiate more complex behaviours, such as cooking or brushing teeth.

In order to determine more complex human activities, the so-called knowledge driven (or specification-based) approaches [93] is more popular as opposed to the data driven approaches. These types of approaches offer the ability to construct high level semantic model and support reasoning/inferencing using the model with multimodal input sensor data. In addition, knowledge-based approaches offer formal structures that can provide semantic meanings to sensor data that enable the construction of high-level semantic activity model to recognize more complex human activities/behaviours.

Among various knowledge-based approaches, ontology-based approaches are ones of the most popular, which are capable of providing a formal specification representing domain knowledge using a set of axioms and constraints [33,51,95,109]. The main drawback of ontology-based approaches is the computationally intensive reasoning process and its lack of support for temporal relations. Further, traditional ontology-based approaches are not able to handle missing sensor data. Some recent research works have been focusing on associating probability and confidence level with ontology to support certain level of uncertainties in sensor data.

4.4.Challenges of WSN-based context-aware systems

As the WSNs technologies are becoming more mature in satisfying various communication requirements, the challenges of WSN-based context-aware systems are shifting towards addressing the utility of the system.

5.1.Vision-based Ambient Assisted Living (AAL)

Ambient Assisted Living (AAL) is to improve the quality of life and maintain independence of older and vulnerable people through the use of technology [27]. State-of-the-art AAL technologies are typically based on solutions employing sensors either embedded in the environment or body worn. The systems are usually costly to maintain, and relatively obtrusive (e.g. sensors set up on every cupboard door) and are highly sensitive to the performance of the sensors. The body worn sensors are quite effective when the user actually wears them, however, it is recognized that user compliance with wearable systems is poor. More recently, the research community and industry have shown a growing interest in video and computer vision based solutions for AAL applications [14,21,43,44,80].

One of the main aims of AAL systems is to provide personal safety to older people. Falling is a major risk faced by older people with one-third of people over 65 falling each year. Therefore the vast majority of vision-based AAL research focuses on fall detection [110]. Zhang et al. [141] present a comprehensive survey of recent proposed fall detection methods, where they divide the existing vision-based fall detection into three categories: fall detection using a single RGB camera, 3D-based methods using multiple cameras, and 3D-based methods using depth cameras. Fall detections using a single RGB camera have been extensively studied, as the systems are easy to set up and are inexpensive. Shape related features, inactivity detection and human motion analysis are the most commonly used clues for detecting falls. The disadvantage of using a single RGB camera is that most of them lack flexibility. Fall detectors using a single RGB camera are often case specific and viewpoint-dependent. Moving a camera to a different viewpoint (especially a different height from the floor) would require the collection of new training data for that specific viewpoint. Single RGB camera based methods also suffer from occlusion problems where part of the human body is occluded by a certain object, like a bed. Another category is 3D-based methods using multiple RGB cameras. The calibrated multi-camera systems allow 3D re-construction of the object but require a careful and time-consuming calibration process. Depth camera based systems [44] share the same advantages with the calibrated multi-camera systems and at the same time do not need a time-consuming calibration step. Depth cameras enable researchers to model fall events under a camera-independent world coordinate system and reconstruct the 3D information of the object.

In a recent paper [80], using a dataset collected from an assisted-living facility for seniors, Luo et al. presented a computer vision-based approach that leverages non-intrusive, privacy-compliant multimodal sensors and state-of-the-art computer vision techniques to continuously detect seniors’ activities and provide the corresponding long-term descriptive analytics. This analytics includes both qualitative and quantitative descriptions of senior daily activity patterns, which can be further interpreted by caregivers. They developed multimodal, privacy-preserving, and easy-to-install sensors and deployed them in a senior home facility. Particularly they utilized the depth imaging and thermal imaging to detect and monitor senior daily activities. Both modalities are privacy-compliant and thus prevent identification of the person in the images. They recorded long-term continuous video data and annotated the data, and showed that, using state-of-the-art Convolutional Neural Networks-based activity classification and detection models, their approach can classify the clinically-relevant activities in real-time as well as to perform temporal activity detection in the long term. Six types of activities: sitting, standing, walking, sleeping, using bedside commode, getting assistance were considered in this work.

5.2.Vision-based patient monitoring

Patient monitoring system is one of the most important devices used in hospitals, which allows the caregivers to regularly observe patients, mainly their vital signs and trends. It is believed that camera-based monitoring can provide more useful information (beyond vital signs) to the caregivers. For example, in clinical environments, abnormal activities in wards such as unbalanced walking, bed climbing, and irregular body posing can lead to serious consequences, for instance falling, pains or injuries. These information cannot be monitored with the existing patient monitor, but can be well detected in the videos. In [31], cameras were installed at intensive care units for observing agitation in the sedated patients. Kittipanya-Ngam et al. [70] studied the use of intelligent video analytic system in monitoring patients in hospital wards. Cameras are utilized for monitoring the activities of the patients and performing better assessments. The proposed system can provide alarming features to draw attentions from the caregivers for detecting falls, risky movements, and irregular actions of the patients. Furthermore, information of the monitored activities is recorded for further studies. They installed cameras in the patient wards with two different views, Bed View and Room View. The Bed View is to capture the activities of the patient on the bed, while for Room View, the purpose is to monitor the activities happen in the common area. The camera of Bed View can help medical staffs to have a better understanding in patient conditions by monitoring the patient 24/7 and providing other information such as bed occupancy and extubation. Apart from activities around their own beds, patients also have activities on the ward corridors. The camera of Room View will provide the data such as accidents and activities inside or outside the wards.

Falls occurring in hospital settings can cause severe emotional and physical injury to patients as well as increase healthcare costs for the hospitals and the patients. Banerjee et al. [13] presented an approach for patient activity recognition in hospital rooms using depth data collected using a Kinect sensor. Depth sensors such as the Kinect ensure that activity segmentation is possible during day time as well as night while addressing the privacy concerns of patients. With the depth sensor, their approach detects the presence of a patient in the bed as a means to reduce false alarms from an existing fall detection algorithm. Their system can reduce false alerts of fall detection such as pillows falling off the bed or equipment movement.

Monitoring neonatal movements may be exploited to detect symptoms of specific disorders. In fact, some severe diseases are characterized by the presence or absence of rhythmic movements of one or multiple body parts. Clonic seizures and apneas can be identified by sudden periodic movements of specific body parts or by the absence of periodic breathing movements, respectively. In [28], motion signals from multiple digital cameras or depth-sensor devices (e.g., Kinect) are extracted and properly processed to estimate the periodicity of pathological movements using the Maximum Likelihood (ML) criterion. Their non-invasive system can help to monitoring 24/7 all the newborns present in a NICU, or even at home. Infants are particularly vulnerable to the effects of pain and discomfort, which can lead to abnormal brain development, yielding long-term adverse neurodevelopmental outcomes. In [119], Sun et al. proposed a video-based method for automated detection of the infants’ discomfort.

5.3.Camera-based physiological measurement

The health condition of a subject is normally assessed by measuring certain physiological parameters of the body, such as body temperature, blood pressure, heart rate (HR), respiration rate (RR), oxygen saturation (SpO2), and so on. These physiological measurements traditionally are performed by attaching sensors in contact with the body (skin), for example, the widely used pulse oximeter. In recent years, camera-based contactless physiological measurement has received much attention because of its easy-to-use and unobtrusiveness.

A number of new methods have been presented [85], where the greatest amount of research has focused on remote photopletysmography (PPG). Photopletysmography is the measurement of light reflected from, or transmitted through, the body, which captures the volumetric flow of blood within the vessels [7,63]. The standard PPG measures the blood volume changes in the microvascular tissue bed beneath the skin in a “contact” manner. Non-contact PPG remotely measures the PPG signals using a camera [120]. Compared to the standard PPG, non-contact PPG have several advantages: firstly, the contact force between probe and skin is not present anymore, avoiding the deformation of vessels walls; furthermore, it is possible to acquire data from larger areas of the skin simultaneously, which allows evaluating the blood circulation in a large area or multi-sites. Verkruysse et al. [128] presented the first example of PPG measurement using a regular, red-green-blue (RGB) camera and ambient light. However, the accuracy of the recovered wave was not rigorously quantified. Poh et al. [99] showed that the PPG signal could be recovered from videos recorded with an ordinary webcam using ambient light.

Different algorithms have been proposed for contactless PPG measurement. The most straightforward method is the spatial averaging; by averaging the pixel intensity values in the region of interest (ROI), the signal to noise ratio (SNR) of PPG can be increased [128,132,143]. A weighted averaging method was proposed in [72], where a higher weight is assigned to pixels that contain a stronger PPG signal and a lower weight to those with weaker signal. Blind Source Separation (BSS) algorithms have been utilized to extract PPG signal from different color channels, for example, Poh et al. used the Independent Component Analysis (ICA) [85,99,100] to extract the pulse signal. To increase the reliability, in [84] they considered a camera with five color channels, red, green, blue, cyan and orange. Similar to ICA, Lewandowska et al. used Principle Component Analysis (PCA) for pulse signal extraction [77]. As a understanding of the algorithm principle behind the non-contact PPG, Wang et al. presented a mathematical model that incorporates the pertinent optical and physiological properties of skin reflection [130]. With this model, a plane orthogonal to the skin-tone can be defined in the temporally normalized RGB space for pulse extraction.

Most of the existing works focus on, as substitute contact PPG, measuring vital sign parameters, e.g., heart rate and respiration rate [11], blood oxygen saturation (SpO₂) [61,132], pulse rate variability [100], and blood pressure [60,140]. A few works have focused on PPG imaging to measure the PPG signals in everything location for, e.g., perfusion monitoring. Hu et al. compared reflection and transmission mode on PPG perfusion maps [144], by acquiring images at wavelengths of 660, 870 and 940 nm on the finger tips. Rubins et al. conducted experiments on elaborating two-dimensional mapping of the dermal perfusion [111]. Kamshilin and colleagues introduced a method for extracting high spatial resolution photoplethysmographic images using the lock-in amplification [64]. They showed qualitatively that it is possible to detect perfusion changes due to scratching of the skin or occlusion of the vascular bed [65].

Thermal imaging is another method for measuring physiological information [104]. It is based on the information contained in the thermal signal emitted from major superficial vessels. This signal is acquired through a highly sensitive thermal imaging system. Temperature of the vessel is modulated by pulsative blood flow. The method is based on the assumption that temperature modulation due to pulsating blood flow produces the strongest thermal signal in a superficial vessel. This signal is affected by physiological and environmental thermal phenomena. Therefore, the resulting thermal signal that is being sensed by the infrared camera is a composite signal, with the pulse being only one of its components. Garbey et al. [50] developed a method to measure cardiac pulse based on a thermal signal emitted from major superficial vessels of the face. Human faces convey not only information about physiological parameters, but also information regarding mental condition and disease symptoms. Certain medical conditions alter the expression or appearance of the face due to physiological or behavioral responses. Thevenot et al. [123] presented a general overview of different works in the literature on using computer vision for health diagnosis from faces.

Most of the car accidents happen because of people’s unsafe behaviors [104], and many aspects of the driver’s conditions may affect the safe driving, for example sleepiness and sleep disorder, clinical depression, alcohol consumption, emergencies (myocardial dysfunction), and even their emotions (anger, excitement, etc.). By continuously monitoring the physiological parameters of the driver, the driving safety could be further improved. Therefore, driver monitoring systems have drawn a lot of attention from the research community and the industry [104]. The use of remote, non-contact camera imaging has been investigated in recent years for in-car health monitoring. For example, Guo et al. [53] presented a camera-based non-contact approach to monitor driver’s Heart Rate Variability (HRV) continuously under real world driving circumstances. Rahman et al. [104] discussed the latest developments in the literature on monitoring physiological parameters using camera and investigate the possibility of using this in driver monitoring domain.

5.4.Challenges and future directions

Nations around the world face a rising demand for costly health care for the ageing population. With the low cost of camera and the maturity of computer vision technology, camera monitoring for healthcare will be further developed with more interests. Although it presents significant promise, camera-based technology has several obstacles to overcome to reach full maturity. The main barriers include users’ acceptability and risk of loss of privacy. The use of cameras would be acceptable in public facilities, i.e. nursing centres and hospitals, and during specific activities or events, such as tele-rehabilitation or safety assessment. However, when used in private spaces (e.g. bedrooms, bathrooms), privacy is a big concern. The use of cameras in private spaces without any privacy protection mechanisms yields in low acceptance by the end user. Hence, maintaining privacy as much as possible is a very important consideration when developing camera-based health monitoring solutions; privacy needs to be protected either by design, or by explicitly taking appropriate measures. One way is to use, instead of normal color camera, the depth sensor and/or thermal sensor, as did recently in [80].

Although we can develop more sophisticated vision-based techniques, it is also worth emphasizing that the vision-based methods are not necessary to be a standalone system for healthcare monitoring. It can be combined with other sensors to improve the robustness and accuracy. For instance, in some AAL applications, audio-video based solutions were proposed, where sound processing and speech recognition are included to help improve the robustness. More recently many multimodal approaches using a network of sensor arrays have been adopted and used.