Improvement of small objects detection in thermal images

3.1Recall on the LIP framework

The LIP framework was first introduced by Jourlin et al. [22, 23, 24], in the context of images acquired in transmission (cf. Annex 1). Let I(D,[0,M[) denote the space of grey level functions defined on the same spatial support D and taking their values in the grey scale [0,M[. If f and gbelong to I(D,[0,M[) and λ is a real number, two operations are deduced from the optical Transmittance Law according to:

The subtraction f⁢⁢g is defined by:

It has been established that the laws and recalled above are internal laws, which means that the resulting functions lie in the space I(D,[0,M[), so that they cannot take values outside the grey scale [0,M[.

For 8-bits digitized images, M = 256 and the grey levels vary from 0 to 255.

Let us recall that the laws and possess strong properties, structuring I(D,[0,M[) as a subset of a vector space. This result gives access to many mathematical tools specific to this kind of space. Note that inside the LIP framework the grey scale is inverted: in such a way, 0 represents the “white” extremity of the scale, i.e., when no object is placed between the source and the sensor.

To complement these benefits, we know that the consistency of the LIP model with the Human Visual System [25] extends its application field to images acquired in reflection, giving the opportunity to interpret such images as a human eye would do.

For the interested reader, Carré et al. [26] have shown that the LIP-addition (resp. subtraction) of a constant to an image (resp. from an image) perfectly simulates a decrease (resp. an increase) of the exposure time or of the source intensity. Concerning the scalar multiplicative law, λ appears as a “thickness” parameter: in fact, given a grey level image f, one can associate to it a virtual half-transparent object producing f in transmission. Then λ⁢⁢f is the image we get by stacking f on itself λ times. This means that the scalar multiplicative law simulates the thickness (opacity) changing of a half-transparent object.

Many new tools have been introduced in the LIP framework, especially in link with the concept of contrast and associated metrics. Here we will limit ourselves to the Logarithmic Additive Contrast [24].

Given a grey level function f and a pair (x,y) of neighboring pixels in the spatial support D, the simplest notion of contrast consists of computing |f⁢(x)-f⁢(y)|. If we replace the classical subtraction by a logarithmic one, we define the Logarithmic Additive Contrast 𝐿𝐴𝐶(x,y)⁢(f) according to:

Due to the scale inversion, 𝑀𝑎𝑥⁢(f⁢(x),f⁢(y)) is always darker than 𝑀𝑖𝑛⁢(f⁢(x),f⁢(y)), so the contrast 𝐿𝐴𝐶(x,y)⁢(f) results in a grey level. Using the definition of the logarithmic subtraction Eq. (3) yields:

Remark 1: The previous formula resembles the classic expression of Michelson’s contrast defined according to:

and:

Obviously, we can observe that the Michelson’s approach overestimates the contrast between two dark grey levels compared to two bright ones with the same difference. A similar effect can be seen with the LAC definition. More precisely an explicit link has been established in [24] between these two contrasts concepts.

Remark 2: If f⁢(x) is noted f and f⁢(y) becomes f+δ⁢f, the above formula shows that the LAC is expressed as M⁢δ⁢fM-f=M⁢W where W represents the Weber constant. This means that the LAC follows the Weber-Fechner law.

3.2Mathematical morphology and Top-Hat LIP

Grey level Mathematical Morphology has been introduced and developed by the Fontainebleau School (Ecole des Mines, Paris) [27]. Given a grey level image fand a binary set S called structuring element, two operations are defined: the dilationf⊕S and the erosionf⊖S of f by S. The structuring element is generally chosen as a disk. For a given point x of D, if Sx represents the disk S centered at x, we have:

Then these two operators are combined to produce the opening(f⊖S)⊕S and the closing(f⊕S)⊖S of f by S which always satisfy the inequalities:

It is well-known that the difference f-[(f⊖S)⊕S] between f and its opening detects all the bright peaks of f. More precisely, this operation consists of moving S inside the subgraph of f and detecting all the parts of the subgraph whereS cannot penetrate.

For thermal images, these peaks correspond to hot objects (pedestrians, cars…). According to the size of S, hot objects, more or less distant from the sensor, will be extracted.

Figure 1.

Illustration of the White Top-Hat. Top: stars picture, middle: difference between f and its opening: f-[(f⊖S)⊕S] with S disk of radius 3, bottom: 𝑊𝑇𝐻t⁢(f) with t= 100.

When applying such a technique to real images, all the peaks are detected, the significant ones and those due to noise. To overcome this problem, the concept of White Top-Hat 𝑊𝑇𝐻t has been introduced [3], in which a threshold t permits to select the peaks higher than t:

The value of t is empirically chosen or preferably determined after studying the noise magnitude. Figure 1 shows the effects of a simple subtraction f-[(f⊖S)⊕S] compared to a 𝑊𝑇𝐻t⁢(f).

In [4] and [5], the White Top-Hat transform is used with a rather empirical modification whose goal is to enhance the contrast of hot objects: the image f is replaced by f+𝑊𝑇𝐻t⁢(f), which means that the peaks height is doubled, with the risk of going out of the grey scale. For this reason, knowing that the LIP addition remains inside the grey scale, we propose to use a LIP addition instead of a classical one, and thus compute f⁢⁢𝑊𝑇𝐻t⁢(f). An example is presented in Fig. 2 for a thermal image of the FLIR dataset.

Figure 2.

Top: image f with pedestrians, middle: f+𝑊𝑇𝐻t⁢(f) showing saturated regions, bottom: f⁢⁢𝑊𝑇𝐻t⁢(f).

As shown in Fig. 2, the classical addition of 𝑊𝑇𝐻t⁢(f) to f clearly generates saturated regions while the LIP addition of 𝑊𝑇𝐻t⁢(f) to f solves this problem. Another significant benefit is that such an enhancement preserves the objects representation in comparison to the learned ones, which will favor their detection.

Note that the Logarithmic Additive Contrast permits to quantify the contrast gain generated by this approach: for each region R of D where the value of 𝑊𝑇𝐻t⁢(f) is not null, the contrast of R with its surrounding background will be increased of the value of 𝑊𝑇𝐻t⁢(f).

In the same way, we define the closing [(f⊕S)⊖S]-f, which detects the dark minima of f. Then the Black Top-Hat 𝐵𝑇𝐻t is defined and is used to accentuate the dark minima by computing f⁢⁢𝐵𝑇𝐻t⁢(f).

Finally, we can summarize the Mathematical Morphological method by performing together the White and Black Top-Hats, which means that the initial image f is replaced by f⁢⁢𝑊𝑇𝐻t⁢⁢𝐵𝑇𝐻t.

Figure 3.

Top-Hat transform with different diameters. Top: 3, middle: 7, bottom: 11.

Figure 4.

Person detection for different values of diameter. Top: 3, middle: 7, bottom: 11.

Now let us consider the multiscale aspect of the problem. Depending on the structuring element radius used in the Top-Hat operators, different information peaks are extracted. The small objects we want to detect in the background of our thermal images have very different sizes according to the scene and to their position and distance from the sensor. It’s understandable that a given structuring element will favor one size of object. Thus, we compute multiple Top-Hat transforms with structuring elements of increasing size, the maximal radius being empirically chosen from detection performance and time constraints. Chaverot et al. [28] proposed a method consisting of executing the neural network object detector on all these Top-Hat transform iterations. The next step is to concatenate the obtained results. The selection of the best bounding-boxes is made with the Non-Maximal Suppression algorithm, NMS [29]. Figure 3 shows the result of the Top-Hat transform for different radii of the structuring element: we can observe that the dark regions like the car windshield become progressively darker while the bright ones (pedestrians) become brighter.

Figure 4 shows the different detections of increasing diameters of the Top-Hat transform structuring element.

3.3Deblurring

As already remarked, thermal images are systematically blurred, in the sense that they look like slightly misfocused images. From an optical point of view, it means that a single bright point is replaced by an Airy disk, whose central part can be approximated by a Gaussian.

During the digitizing step, if the Airy disk diameter is larger than the size of a pixel x, only a percentage of the light particles (photons for example) destinated to x will really contribute to its grey level f⁢(x). To correct this kind of blur, many solutions have been explored. One of the oldest was presented by Pratt ([30], Section Edge crispening) as an application of the Laplacian operator and consists of adding to the initial digitized image f its Laplacian 𝐿𝑎𝑝⁢(f) defined as a 3 × 3 convolution. Observing that such a method does not consider the blur level, Jourlin proposed a more accurate technique [31] in which f is replaced by fα=f+α⁢𝐿𝑎𝑝⁢(f), the value of α being chosen to maximize a quality parameter computed on fα. This approach is very efficient as long as the Airy disk is not larger than a 3 × 3 mask.

In the literature [32], a resembling approach, commonly referred as the Unsharp Masking (UM) algorithm, consists of computing fλ=f+λ⁢z, where fλ and f denote respectively the enhanced and original images and zthe correction component. This correction component can be a Laplacian operator. Nevertheless, for computational purposes, we consider the similarity of the Airy disk with a Gaussian function. If G⁢(f) represents the convolution product of f by a Gaussian filter G, the correction factor z is computed as follows: z=f-G⁢(f).

Figure 5.

Unsharp masking. Top: initial thermal image f, bottom: deblurred image f1.

An example of such a deblurring is shown in Fig. 5 with λ= 1, which means that f1=2⁢f-G⁢(f).

3.4Super resolution

In the case of an already trained network, modern CNN architecture permits to increase the input image resolution without having to retrain it. Increasing the network resolution has the advantage of detecting smaller targets, despite an increase of the processing time. Considering the same original dataset, in this approach, the input images require an upscale too. A bilinear or bicubic interpolation is usually used for this task. But in our situation, image Super Resolution (SR) appears particularly interesting.

SR algorithms aim at increasing images resolution while keeping precise and sharpened details in comparison of classical interpolation methods generating blur and approximated information (e.g. bicubic interpolation (BC)). Generally, SR methods provide an increase in resolution by a factor × 2, × 3 or × 4. For example, applying a SR algorithm to a target of size (m,n) results in a (2⁢m,2⁢n) resized target for a factor × 2 and a (4⁢m,4⁢n) one for a factor × 4. We assume that an object detector failing at detecting too small targets can retrieve them if they are properly resized.

Recently many deep learning-based SR methods have been developed and achieve new state-of-the-art performances. We can refer to the following reviews presenting recent SR works [33, 34, 35]. In our experiments we use the CARN model (CAscading Residual Network) [36], based on a residual network and implementing a cascading mechanism. This network proposes a satisfying image quality and is quite efficient in term of computational complexity thanks to its lightweight design.

In the learning phase, couples of low resolution and high resolution images are required. High resolution images correspond to the original images of the training dataset (with the native resolution). Low resolution images are obtained by reducing the images size by factors × 2, × 3 and × 4. The network is trained to retrieve high resolution images from their low-resolution version. Since three resized factors have been used in the training phase, we can choose between these different factors when executing the network. Figure 6 proposes a comparison between resized images by bicubic interpolation and by our specialized SR network. The SR image looks slightly sharper than the interpolated one.

Figure 6.

Super resolution applied on thermal images. Top: bicubic interpolation. Bottom: SR × 2.

A comparison between SR and bicubic upscaling is proposed in Table 1. On some images of the FLIR dataset, PSNR and SSIM metrics are evaluated between Ground Truth (GT) and upscaled images. GT images correspond to original FLIR images, the upscaled ones are obtained by downscaling the GT and then upscaling them by SR or bicubic interpolation. On every comparison, Super Resolution gets the best scores.

3.5Combined algorithms

We have also explored combinations of our previous proposed methods. We focused on combinations of SR with UM (Unsharp Masking) and SR with Multiscale Top-Hat transform. As the SR multiplies the number of pixels in each direction, we adapt the radius of the structuring element used in the Top-Hat transform and the neighboring region size of the Unsharp Masking to consider the novel targeted size. Typically, we double the size for a SR × 2.

The combination of UM and Multiscale Top-Hat transform has been tested, as the first one enhances the information peaks to be detected by the Top-Hat operations. However, such a combination yields poorer detection performance than using only one of the previous exposed methods and will not be further discussed in this paper.

4.1Materials

4.1.1FLIR dataset and validation subset selection

The FLIR ADAS dataset [37] is a dataset of thermal images proposed by the FLIR company, a thermal image sensor manufacturer. This set is constituted of 10228 images with a resolution of 640 × 512 pixels and includes 80000 annotations. Images have been acquired in the Santa Barbara region in California in various situations of daytime and weather. Thermal images are proposed in two formats, RAW 14 bits images and 8 bits preprocessed images. The last ones are issued of FLIR own enhancement and domain adaptation algorithms. We choose to use the RAW thermal images provided by FLIR, normalized into an 8 bits greyscale with a dynamic expansion centered on a fixed mean m= 0.5 and standard deviation of σ= 0.25. The motivation to use RAW images instead of the already transformed ones is to avoid artifacts generated by FLIR’s own enhancement algorithm and to apply a custom enhancement algorithm. In Fig. 7, a FLIR enhanced image is compared to our normalization combined with Unsharp Masking: this last approach preserves the details of the persons while the FLIR transformation saturates the information.

The FLIR ADAS dataset provides annotations for four classes: persons, cars, bicycles, and dogs. Table 2 presents the classes distribution.

Table 2

FLIR ADAS classes distribution along train and validation split

	Persons	Bicycles	Cars	Dogs
Train	13725	3297	36642	178
Val.	4955	441	5209	12

Figure 7.

Top left: 14 bits RAW image, top right: the proposed normalization, bottom left: the proposed normalization and unsharp masking, bottom right: FLIR enhanced image.

In this paper, due to the large unbalance between the classes, we decided to focus only on the two following classes: persons and cars.

The FLIR ADAS dataset is not perfect. The annotations of small objects are not present. It can be explained by the difficulties to annotate them due to the poor quality of thermal images. To obtain accurate scores of detection performance, 693 images from the validation dataset were chosen and reannotated. One hundred of these images were chosen because of their interesting scene, with vehicles relatively far, and pedestrians crossing the street in the background. The rest of the images were randomly picked. Full list of images and annotations files are freely available on GitHub [38]. The selected validation images contain 5318 annotated persons and 3206 annotated cars. Figure 8 shows the difference between the original annotation and our novel annotation. Table 3 shows a comparison of the number of labeled objects before and after our reannotation. The number of small bounding boxes has doubled, while the count of boxes in other sizes remained relatively unchanged.

Table 3

Bounding boxes areas distribution along our selection

	Small	Medium	Large	Total
Original	2378	2577	538	5493
Updated	5669	2676	546	8891

Figure 8.

Comparison between FLIR annotation (top) and the new one (bottom). Red boxes are labeled cars, Green boxes are labeled pedestrians.

4.1.2Fine tuned YOLOv4

YOLOv4 is an object detector proposed by Bochkovskiy et al. [39] leveraging multiple features to obtain the best trade-off between inference time and detection accuracy. The authors aim to design a detector suitable for systems production and optimizable for parallel computation. This detector is built with CSPDarknet53 [40] as backbone due to its state-of-the-art classification results on MSCOCO [41]. CSPDarknet53 is associated with a Spatial Pyramid pooling module, increasing the separation of significant features without reducing the processing speed. The path-net aggregation neck of the detector is PANet [42], shortening the information path between lower layers and topmost feature. At last, the detector head is the same as YOLOv3 [43], anchor based.

YOLOv4 also uses state-of-the-art training strategies and data augmentation methods, increasing detection performance and overall robustness.

The network architecture is presented in Fig. 9 and a performance comparison chart given in [39] is shown in Fig. 10.

Figure 9.

YOLOv4 architecture [39].

Figure 10.

Performance comparison chart.

In our experiments, we use a fine-tuned YOLOv4 on the FLIR ADAS dataset [28].

The fine-tuning was made with the following parameters:

• • Resolution: 640 × 512 pixels

• • Optimizer: Stochastic Gradient Descent

• • Learning Rate: 0.001

The optimizer and learning rate were selected in accordance with the original publication of the network.

The train and validation split provided by FLIR was kept. The mean Average-Precision, mAP, score was computed on the validation set at each epoch for a minimum of 60 epochs, the model obtained is the one with the best score along the training. This model has the state-of-the-art detection performance on the FLIR ADAS Dataset.

4.2Experimental settings

For the YOLOv4-based network, the Unsharp Mask is computed using a Gaussian filter with a standard deviation σ= 1.4, corresponding to a neighboring region of diameter k= 3. On the other hand, for the EfficientDet-based network, the Unsharp Mask uses a Gaussian filter with a standard deviation σ= 2.3, corresponding to a diameter k= 11.

These diameters have been set thanks to a sensitivity analysis for the impact of the kernel diameter of the Gaussian filter. This analysis is presented in Figs 11 and 12, showcasing the effects of different diameters on the results.

Figure 11.

Sensitivity analysis: influence of the Gaussian filter diameter of the UM method with the YoloV4 fine-tuned model.

Figure 12.

Sensitivity analysis: influence of the Gaussian filter diameter of the UM method with the EfficentDet-D3 fine-tuned model.

When the Unsharp Mask is applied to the Super-Resolution (SR) × 2 method, the value of σ is increased proportionally to the doubled value of the filter diameter.

The Multiscale Top-Hat transform employs a circular structuring element with an odd diameter ranging from 3 to 13. When used in combination with the SR × 2 method, the following diameters are used: 7, 11, 15, 19, 23, and 27. The chosen range of values for the kernel diameter in the Multiscale Top-Hat transform is intended to specifically target the enhancement of small objects contrast. By limiting the diameter within the range of 3 to 13 for the Multiscale Top-Hat transform and using a standard deviation within the specified range for the Unsharp Mask, the focus is on enhancing the visibility and brightness of small objects in the images. It is worth noting that larger values for the kernel diameter can result in more significant alterations of the original image and may require higher computational resources. Therefore, the chosen values strike a balance between achieving contrast enhancement for small objects and avoiding excessive image distortion and excessive computational overhead.

For the concatenation and selection of the best bounding boxes in the Multiscale Top-Hat method, a Non-Maximum Suppression algorithm [29], NMS, is applied with a threshold of 0.45. This threshold value is commonly used when inferring a detection network and helps eliminating redundant bounding boxes.

4.3Results and discussion

In our study, we present the results in Tables 4–7, which provide comparisons of Average Precision (AP) and mean Average Precision (mAP). The computation method for AP is defined in the MS COCO Dataset [41]. Specifically, we used the AP50 score, which corresponds to the AP computed at an IoU (Intersection over Union) threshold value of 0.5. To further analyze the performance, we also considered the classification of AP50 scores into Small, Medium, and Large categories as defined in [41]. These categories are used to assess the performance of the detection model across different object sizes.

The baseline score is obtained with the fine-tuned model presented in [28] on our validation subset. The four presented methods are: MTH (Multiscale Top-Hat), MTH LIP, UM (Unsharp Masking deblurring), SR (Super Resolution). For a comparison purpose, we have included results obtained from two additional image variations. The first variation involves using images enhanced automatically by the FLIR algorithms (FLIR Enhanced). The second variation involves resizing the images by a factor of two using Bicubic Interpolation (BC).

For each considered detector, the LIP version of the MTH has outperformed the classical one. The UM and the SR × 2 method permit to produce better results than the baseline. It is worth noting that the observed improvement in performance is particularly notable for the Small objects category, which aligns with our specific objective. This outcome demonstrates the effectiveness of our approach in addressing the challenges associated with small objects detection.

However, it is important to mention that when employing Bicubic interpolation or Super-Resolution methods, there is a decrease in the AP50 scores for the Large objects category. This drop in performance for larger objects does not significantly impact the overall mAP score, which considers the performance across all objects categories.

This discrepancy suggests that while the BC and SR techniques impact the detection of larger objects, the overall detection performance, as measured by mAP, remains positive. It highlights the trade-off between enhancing small objects detection while potentially affecting the detection accuracy of larger objects when using these methods.

The SR × 2 method outputs better performance than the BC x 2 one, suggesting than an image obtained via an SR network gives better detection performance.

The following combinations have been tested: SR × 2 with UM and SR × 2 with MTH LIP. Both improve detection performances. Compared to the Baseline, the mAP gain is above 5% with YoloV4 and 11% and 13% with EfficientDet-D3, and we remark that SR × 2 with MTH LIP increases the AP50 of Persons of near 7% and 13% respectively, resulting in the best observed improvement. In both cases: single method and combined ones, the MTH LIP appears essential.

In our experiments, using FLIR images as training dataset, the scores obtained by EfficientDet-D3 are globally lower than those obtained with YoloV4 while these networks show more similar performances on the COCO dataset. These performance differences can be explained by the dataset sizes (328,000 images in the COCO dataset vs 10,228 in the FLIR dataset) and the ability of YoloV4 to perform new augmentation techniques [39]. YoloV4 appears more adapted to reduced datasets.

The execution times of fine-tuned models and preprocessing algorithms are presented in Tables 8 and 9, respectively. Additionally, the frames per second (FPS) obtained with the full pipeline using YoloV4 or EfficientDet are also reported. These times were obtained using an Intel Core i9-10850k CPU and an NVIDIA RTX3090 GPU, with Python, PyTorch 1.8.2 and CUDA 10.2. The YoloV4 architecture is more parallelizable than the EfficientDet one, which explains the different inference times measured. The UM algorithm requires a very low execution cost, making it an attractive option. The MTH LIP is significantly slower than the original MTH, however, parallelizing the algorithm and implementing it in CUDA would accelerate it. Although the BC2 appears to be very effective, the increasing inference time of the neural network for a doubled resolution should not be ignored. In the end, the choice of the preprocessing will depend on the initial performance level of the network, the performance requirement, and the available processing time. In our case, network quantization can be a valuable technique to reduce network inference computation, as studied by Wu et al. [47]. By applying network quantization, we can decrease storage costs and reduce computation time associated with network inference.

It should be noted that the training of the EfficientDet detector was not optimized for the best performance, the purpose of this example is to show that our method can be applied to a trained detector and achieve better performances, even when the training is not carried out in the best possible manner.

Considering the scores presented in Tables 4 and 6, we notice that with both networks YoloV4 and EfficientDet-D3 our image pre-processing techniques provide a significant performance gain. With YoloV4, from our baseline of mAP 74.80, applying SR x 2 and MTH LIP on input images leads to a mAP score of 80.11. Using the same pre-processing technique with EfficientDet-D3, from a baseline of mAP 47.49, the performance reaches a mAP of 60.14.

These results demonstrate the ability of the proposed pre-processing techniques to improve the detection performance of an already trained network. Despite the performance gap between the two detectors, the results demonstrate the effectiveness of our proposed methods, and shows the generalization possibilities to other datasets and tasks.