Efficient surface defect detection in industrial screen printing with minimized labeling effort

1.1Machine vision inspection process

In general, machine vision inspection can be roughly divided into three main stages: defect visualization, preprocessing, and inference. Based on the optical surface properties of the sample under investigation, appropriate optical components must be determined. These components include cameras (sensors incl. optics), illuminations, and filters, which are used to visualize defects and reduce the prominence of unimportant features. Thus, the objective of defect visualization is to maximize the contrast of imperfections on product surfaces in digital images, making them more easily identifiable and analyzable. Determining suitable optical components typically requires extensive laboratory experiments and domain knowledge. General approaches for automatically characterizing defect visibility, applicable to various surfaces and defect textures, remain an active area of research [19, 20, 21].

In addition to the common RGB and monochrome sensors in the visible range, sensors operating in the ultraviolet (UV) or infrared (IR) ranges can offer advantages for specific features. For instance, UV sensors (200–400 nm) reveal fine scratches on polished surfaces that are barely visible to the human eye (e.g. Sony’s IMX487 [22]). Shortwave infrared (SWIR) sensors (900–1700 nm) are increasingly applied in the electronics and semiconductor industries to uncover subsurface defects [23]. Multi- or hyperspectral cameras, which combine different spectral bands (e.g. visible and IR), reveal spatial physical and chemical properties of the samples being examined [24, 25]. These imaging techniques are particularly valuable in the food, waste management, packaging, agricultural, and pharmaceutical sectors [26, 27]. However, exploiting additional spectral bands is accompanied by an increased workload for data processing. Besides the selected spectrum, the optimal optical modality is determined by the illumination conditions, including the illumination characteristics (e.g. direct, diffuse, structured) and its position relative to the surface and sensor (bright field, dark field or transmission).

In addition to the defect visualization capabilities, factors such as system integration complexity, data processing bandwidth and software interoperability, are crucial in selecting appropriate hardware for industrial applications.

Image preprocessing prepares the captured data for inference with the selected defect detection methods. This stage may involve tasks such as image registration, masking, resizing or data normalization, to name a few. Finally, inference is used for the decision-making, classifying the inspected product as normal or defective.

1.2Industrial defect detection methods

With the rise of affordable computing power, deep learning-based research has gained significant momentum in machine vision tasks. Deep convolutional neural networks (DCNN) have shown superior performance over traditional defect detection methods that rely on manual feature engineering [28]. The performance of deep learning methods typically scales with the amount of available training data. However, collecting large quantities of labeled data is labor-intensive and often impractical for many industrial applications. In the context of surface defect inspection using supervised neural networks, this is a major limitation, as an extensive labeling process is required for each new product type to meet inspection standards. Consequently, current research focuses on semi- or unsupervised defect detection methods [29, 30], which require minimal or no defective samples for training and are therefore of particular interest for industrial applications. By modeling the underlying data distribution of fault-free (normal) samples, unsupervised methods overcome possible generalization problems of supervised methods.

Since the publication of industrial defect detection datasets such as MVTec [31], several anomaly detection methods have emerged. These unsupervised methods can be broadly categorized into representation-based [32, 33, 34, 35], generative model-based [36, 37, 38, 39, 40, 41, 42], and flow-based [43, 44] approaches. Representation-based methods compare test data features with learned normal representations to measure feature similarity or distance. Flow-based methods map feature distributions to multivariate Gaussian distributions using normalizing flows, with deviations indicating anomalies. Both approaches use DCNN feature extractors pretrained on large datasets like ImageNet [45]. These methods are often memory and computationally intensive due to their architectures and algorithms, such as k-nearest neighbor. Furthermore, the feature extractors are biased towards the dataset used for pre-training, which leads to performance degradation in case of significantly different examined data distributions.

Generative models are designed to reconstruct normal data, failing to properly reconstruct defective regions resulting in anomaly scores. Despite progress with autoencoders [36], generative adversarial networks (GANs) [46, 38, 39] and denoising diffusion models [40, 41, 42], challenges persist in overcoming reconstruction limitations for fine-grained patterns as well as computational efficiency.

Additional approaches include synthesizing defective data samples for self-supervised pretraining or data augmentation [47, 48, 49, 50, 51]. However, GAN-based synthetization tends to generate simple defect structures and struggles with complex patterns. In addition, these methods rely on large datasets including defective samples for the initial training process. Recent research on few-shot generative models, including diffusion-based approaches, aims to address these issues [52, 53, 54, 55].

Given the specific data distributions and detection tasks in industrial applications, specialized methods are crucial. The high permissible variability in complex design patterns of screen-printed products and their diverse product portfolio demands robust and adaptable methods. In addition, short inference times are mandatory in order to achieve the required process cycle times.

1.3Related work

Ongoing research on defect detection of screen-printed products is being increasingly applied in the electronics industry. In Zhao et al. [56], the screen-print of batteries is inspected using a multi-level block template matching and k-nearest neighbor method. The presented inspection system enables the detection of defects, such as blurred prints, local defects or scratches of the printed product logo, QR code and fabrication number. Further work presents an automatic inspection system for surface defect detection of screen-printed mobile phone back glasses [57]. A dual brightfield imaging system is demonstrated for defect visualization. It consists of a coaxial bright field and a low angle bright field illumination, enabling the visualization of defects such as scratches, dents and discolorations. Defect detection was performed with a symmetric semantic segmentation network trained in a supervised manner. The training dataset consisted of 34 550 images (6742 defectives), achieving an average test precision and recall of 91.8 and 95.3%. Another inspection solution for mobile phone cover glasses is presented in [58]. The system adopts backlight imaging in combination with a segmentation method trained in an adversarial manner utilizing a novel data generation process. A further defect detection method applied to a screen-printing process is based on an optimized U-Net++ [59] architecture, which is described in [60]. To enable accurate detection of small defects in relation to the product size, only image patches were evaluated rather than the entire image. The visualization of the defects was done with a white backlight and a blue incident illumination. Using the patch-split method and a customized loss function, a dice score of 0.73 was achieved. Gafurov et al. [61] investigated smearing effects of screen-printed lines using deep neural networks (DNN) and CCD cameras, installed subsequent to the screen-printing process. For this purpose, a screen-printing mask was designed containing different line widths and spacings as well as a variation of squeegee directions. Using an adapted U-Net architecture, it was possible to detect smearing defects in various printing conditions.

Commercially available automatic visual inspection systems are known in the printing, glass and weaving industries [62, 63, 64, 65, 66, 67, 68, 69, 70]. However, inspection solutions for defect detection in the field of screen-printing are rather limited. The company OMSO [71] offers a product for optical inspection of decorations on cylindrically shaped objects such as bottles, tubes and jars. Cugher’s glass inspection system [72] enables the detection of defects on the screen-print designs of glass panels. Keko Equipment Ltd. [73] offers an automatic inspection system to inspect prints on multilayer green ceramic productions. The inspection software leverages e.g. golden template comparison applicable for max. inspection areas of 220 × 220 mm at a resolution of 10 μm. An integrated inspection system following a screen-printing process is offered by Sakurai Graphic Systems Corporation [74] by means of their SI Inspection Unit. Thereby, two line scan cameras in incident light illumination compare the screen-printed pattern to a master reference sample.

1.4Contributions

The aforementioned studies and commercial automated inspection systems mostly contain inspection solutions for printed product designs such as logos and labels, showing clearly defined geometries and image features. Due to non-complex print designs, e.g. by using only a few print layers, the spectrum of possible defect causes and subsequent diverse defect appearances is reduced, which limits the effort of defect visualization. Frequently used deep learning-based segmentation models rely on pixel-wise labeled ground truth masks for supervised training and are therefore dependent on the amount and quality of labeled data.

The products studied for the given publication are designed for use as decorative patterns in a variety of applications, including products used in the automotive industry. In order to meet the customer’s needs and requirements, complex designs are developed and manufactured under high quality standards. To achieve the desired visual impression of the decorative pattern, numerous manufacturing steps are necessary, resulting in a complex multilayered design. Therefore, the aim of this work was to develop an automatic visual inspection system that inspects decorated foil plates for production-related surface defects. Generally, the defects appear small in relation to the sample size being examined. The developed optical modalities must be able to display the different defect classes with sufficient contrast. Given the large product portfolio, the adaptability of the system to new product designs is of great importance. In addition, the applicability of automated visual inspection in the production line should be ensured with regard to important process requirements such as cycle times. Due to the high labor involved in data acquisition and labeling, defect detection methods that provide sufficient detection performance with as little labeling effort as possible are emphasized. Furthermore, they must be capable of handling the allowed product to product variability.

Currently, there is no available inspection system for automated full-surface defect detection of decorated foil plates, accounting for allowed design and product variability and adaptability. It has to be mentioned that this publication builds upon the research work presented at the ASPAI 2022 [75]. Optical modalities were introduced that enable the visualization of production-related defects with sufficient contrast. Therefore, laboratory experiments were conducted to analyze various design patterns of different products. The possible integration of investigated optical modalities into the production line as part of an inspection approach was outlined. By assigning the detected defect classes to the individual production steps, deviations in the production process will be detected at an early stage.

Thus, the main contributions of this work can be summarized as follows:

Section 2 illustrates the structure of decorated foil plates and briefly describes the manufacturing process. Frequently occurring defects are visualized and the formation process of selected ones is described. In Section 3 experimentally explored optical modalities are specified, followed by an introduction of the inspection system and its underlying procedures in Section 4. Section 5 presents the investigated defect detection methods. Section 6 gives an evaluation of their defect detection performance as well as inference speed and overall inspection time. Section 7 provides a summary of the key findings and an outlook for future improvements.

2.1Defect formation process and defect classes

The complex manufacturing process results in a large number of possible defect causes. Basically, defects can occur in every manufacturing step, which cumulatively affect the final product. Understanding the origin of defects and their visual appearance is of central importance for optimizing the quality standards in the manufacturing processes and, in the event of their occurrence, for taking corrective action. In this work, possible process-related surface defects are investigated. In the case of surface defect detection, a defect can be generally described as any sufficient deviation from the normal sample, considering the allowable product variability. Typical defect classes include e.g. printing defects, inclusions, mechanical deformations, scratches, smears, squeegee strokes, pinholes, dust and misregistered control markers. On the right side of Fig. 1, two image patches on a structured decorative pattern are illustrated. The defects appear small (approx. 0.07 mm² on the upper right defective patch), i.e. only a few pixels in size, relative to the product size of up to 10⁶ mm². A further characteristic is the high permissible design variability of the structured patterns. This is evident when comparing the variance in contrasts of the patches mentioned above. Depending on the location of occurrence and product design, these defects can also be defined as weakly contrasted.

Figure 2.

Visualized defective patches of selected production-related defect classes captured by means of the customized optical modalities. Each row corresponds to a modality, from top to bottom: Line Scan Modality 1, Line Scan Modality 2, Area Scan Modality. Depending on their spatial appearance, defects can be divided into point defects such as inclusions (1A), screen or print defects (1B), scratches or dots (3A/3B) and pinholes (2A), or area defects such as pattern misalignment (1C/2C), inhomogeneities (2B) or squeegee strokes (3C).

Figure 3.

Relative positioning of the camera sensors and illuminations regarding the utilized optical modalities: Line Scan Modality 1 (LSM-1), Line Scan Modality 2 (LSM-2) and Area Scan Modality (ASM), as presented at the ASPAI 2022 [75].

Prior to each print cycle, the new print layer is precisely aligned to the existing layers. Any misalignment of individual print layers during this registration process will be visible in the printed pattern by a so-called pattern misalignment, which is apparent across the entire surface (Fig. 2, 1C/2C). Deviations during the ink application process, e.g. regions with too less ink application, lead to pinholes or inhomogeneities (Fig. 2, 2A/2B). Inhomogeneities are print layers with too low optical density and high variance in color values. Pinholes, in turn, appear as dot-shaped holes in the print pattern. Impermissible holes or closed meshes in the stencil of the screen can lead to screen and print defects (Fig. 2, 1B). Typical inclusions in individual ink layers, such as dust and fibers (Fig. 2, 1A), are caused by impurities in the process environment and electrostatic charge on the foil plates. Due to an electrostatic interaction with charged particles in certain inks, static splashes or stains may also occur. Automatic or manual product handling can cause scratches or dots (Fig. 2, 3A/3B), as well as two-dimensional mechanical deformations, within the topcoat layer. Changes in the uniformity of the squeegee pressure and the dwell time of the ink on the screen can lead to a variable ink application. These so-called squeegee strokes (Fig. 2, 3C) are characterized as area defects and are affected by the printing direction.

3.1Line scan modality 1

LSM-1 consists of an RGB linescan camera and a high intensity LED line bar (white, 6200 K). The illumination is equipped with a special lens and light amplifier foil to ensure the most directional and brightest illumination in the focal zone. The optimum distance, determined by the optical characteristics of the line bar, is approximately 50 mm above the sample’s surface. The relative arrangement of light source and camera, enable a dark field illumination. The camera is placed planar to the sample’s surface and the angle of incidence of the illumination is chosen as steep as possible w.r.t. the horizontal plane. This positioning avoids strong shadowing in complex decorative patterns showing a 3D effect. Patches 1A–1C in Fig. 2 were recorded by means of this setup. Small punctual defects such as screen defects or inclusions that stand out only slightly from the background, are displayed in good contrast. Furthermore, area defects such as the pattern misalignment of an entire print layer, occurring as semitransparent white overlay in patch 1C, is clearly pronounced. This setup is applicable for defects affecting the decorated pattern like slurred prints. In addition, it addresses “sawtooth” defects, defined as continuous eroding at patterned edges, as well as misregistered control markers.

3.2Line scan modality 2

LSM-2 similarly utilizes an RGB line scan camera and a high intensity LED line bar. The illumination is placed planar and opposite to the camera aligned to its optical axis. As shown in Fig. 3, this setup allows transmission measurements of the investigated sample. As in LSM-1, the optimal distance is determined by the optical characteristics of the line bar (approx. 50 mm behind the sample) to achieve the highest possible illumination intensity. A planar alignment of sensor and illumination to the sample’s surface is mandatory to reliably investigate thicker layers on large-sized samples, mitigating geometrical influences on the optical path. As in the LSM-1 setup, the camera distance is determined by the demanded maximum FOV as well as the required object pixel size and can be greater than 1000 mm dependent on the sensor design. In addition to defects such as pinholes or pattern misalignment (Fig. 2, 2A/2C), it is also possible to display unwanted inhomogeneities in semitransparent colored print layers (Fig. 2, 2B). Line scan cameras in combination with high intensity line bars are generally the appropriate choice for the dynamic inspection of flat surfaces, as they are capable of capturing high resolution images at high measurement speeds, regardless of the sample size in the transport direction. However, the experiments conducted with the line scan camera revealed that the detection of defects on the transparent top layer was not satisfactory.

3.3Area scan modality

To overcome above mentioned limitations an optical modality, consisting of an area scan camera and a light bar aligned in direct reflection, was designed (ASM in Fig. 3). Therefore, the LED light bar (white, 6200 K) is placed as far away as possible from the specimen’s surface to create a large optical lever with respect to the monochromatic camera sensor. As shown in Fig. 2 3A–3C, this illumination method produces a bright area of direct reflection in the center, which decreases and fades out to the margins. Defects such as squeegee strokes or smears are only visible with high contrast in this transition area of reflection (Fig. 2, 3C). In general, bright-field images differ significantly from dark-field images in the LSM-1 and reveal defects in the transparent top layer, such as scratches and mechanical deformations (Fig. 2, 3A/3B). The majority of defects in the transparent top layer are only visible using this modality.

Utilizing all three modalities it was feasible to visualize the required production-related defect classes with sufficient contrast within the range of a few pixels in extension and minimum object pixel sizes of approx. 75 μm. The different object pixel sizes are clearly apparent in Fig. 2, as patches 3A–3C map a larger FOV at the same pixel size (256 × 256 px) as patches 1A–2C captured by LSM-1 and LSM-2.

4.1Inspection procedure

As shown in Fig. 4, the inspection system demonstrator consists of three measurement chambers, respectively one for each modality LSM-1, LSM-2 and ASM, which are arranged in sequence. Each measurement chamber is optically shielded to avoid both ambient light and unwanted reflections from different chambers. Regarding the above stated system requirements as well as required optical modalities, suitable hardware components had to be selected. The hardware components of the measurement chamber LSM-1 and LSM-2 consist of commercial 16k RGB line scan cameras including optics and commercial high-power LED line bars according to Section 3. Due to the large required FOV of 1200 mm, 4 side by side monochrome area scan cameras (2.2 MP) incl. optics in combination with a high-power bar light are mounted in ASM. During each measurement cycle, the sample is manually placed on a conveyor belt and sequentially transported through all three measurement chambers. An installed rotary encoder generates trigger signals that enable distortion-free image acquisition at different conveyor speeds. Optical sensors detect the onset of the sample’s surface and thus start image acquisition. Furthermore, the sensors characteristics were calibrated for a variety of decorated surfaces. The system parameters such as i.e. conveyor speed, illumination characteristics and sensor data are centrally controlled by means of a Programmable Logic Controller (PLC) and can be adjusted via an Human Machine Interface (HMI) panel. The inspection software operates on a distributed infrastructure. The computing unit consists of two computers, each with an NVIDIA GPU (GeForce RTX 2080 Ti resp. RTX 3090), a multicore processor and frame grabbers for the cameras.

Figure 5.

Illustration of the patch extraction process in the defined ROI. Overlapping patches (blue squares) are extracted within the entire ROI-area (green), ensuring a minimum covered sample area at the borders.

4.2Data processing

The images of the LSM-1 and LSM-2 measurement chambers consist of 16384 × 12800 px for each color channel at an FOV of 1200 mm in sensor direction. The images captured by ASM even contain 8192 × 65536 px due to the image stitching procedure. This results in a data amount of 600 MiB per LSM-1 and LSM-2 chamber as well as 512 MiB in case of the ASM chamber. In total, approx. 1.7 GiB of image data are processed per measurement cycle. Given the required cycle times ranging from 15–30 s, this represents a major challenge in terms of data processing. For this purpose, a memory-optimized and parallelized data processing pipeline was developed. Thereby, the measurement data, starting from the raw image of the camera up to the final inference decision, are processed in parallel. Key processes include image acquisition, registration, segmentation, inferencing, post-processing and saving. A key feature of the data processing pipeline is that it avoids the evaluation of the entire image at once. Instead, as part of the preprocessing, overlapping image patches of size 256 × 256 px are extracted within masked image regions that are relevant for defect detection. This results in localized patch-wise defect detection. As shown in Fig. 5, overlapping patches (blue squares) are extracted in the so-called “region of interests” or ROIs (green border) of the raw image in the form of a grid covering the entire samples area. These regions, predefined by domain experts, define the product surfaces subject to inspection. A zoomed fault-free patch is visualized in the upper right corner. The overlapping patches with strides of 120–160 px ensure that a potential defect is completely covered at least in one patch, avoiding overlooking at borders. This results in 170–300 patches per ROI. Depending on the product size and the number of ROIs to be inspected, there are several thousands of patches (e.g. 3400–6000 patches for 20 ROIs) to be evaluated per image and measurement chamber.

Figure 6.

Excerpts of masked sample images, acquired by means of the three measurement chambers: LSM-1, LSM-2 and ASM within a sequential run.

4.3Calibration and image acquisition

The calibration of the measurement modalities included optimizations of the rotary encoder settings, photo responsive non-uniformity corrections, fixed pattern noise corrections as well as white balancing. Depending on the modality, minimum exposure times of 100 μs were selected to achieve the desired contrasts and limit motion blurring. The MTF50 – value of LSM-1 could be estimated with the help of an ISO Standard 12233:2000 calibration chart and a self-implemented slanted edge method as described in [79] with 5 lp/mm. With the help of the line scan cameras in the measurement chambers LSM-1 and LSM-2, it was possible to capture samples with FOVs of up to 1200 mm without gaps. The acquisition of the entire FOV in measurement chamber ASM is only possible by software stitching of the individual image frames. The difference of LSM-1 and LSM-2 compared to ASM in the form of a stretched geometry of the design, is evident in Fig. 6. The frame rate was chosen as high as necessary (up to 40 frames per second) to avoid possible overlooking of small defects in the aforementioned transition area of direct reflection.

5.1Supervised oversampling method

A widely used approach in machine learning to compensate for imbalanced data distributions are resampling techniques such as random undersampling or oversampling [80, 81]. In the case of an imbalanced data distribution of majority and minority classes, the samples from the respective class are randomly eliminated (undersampling) or copied (oversampling) to create a revised balanced dataset. In the course of this work, a scalable patch-wise oversampling method is introduced that enables efficient use of scarce and imbalanced data available. The following steps are necessary in providing fault-free (majority class) and defective data (minority class):

Parameters such as the minimum ROI area covered per extracted patch and the maximum distance of random translations during augmentation, determined by defect type and patch size, must be carefully chosen. It is crucial to ensure that product surfaces are sufficiently represented and defects are covered after extraction and augmentation, to avoid e.g. generating defective patches that miss defective areas.

The samples used are pre-sorted by a domain expert prior to image acquisition, with separate samples for fault-free and defective data. This task eliminates any unwanted correlation between defective and fault-free patches in the subsequent data set generation. Thus, the labeling effort is limited only to the defective data at known regions, since the extraction of fault-free patches is integrated into an automated process (step 1 to 5). Furthermore no elaborate pixelwise labeling of ground truths as in in segmentation based approaches is required. By extracting patches at random positions, the original dataset can be exploited as much as possible (several 100 000 patches from a few acquired images with large FOVs). Moreover, it is theoretically possible to collect an endless stream of fault-free patches. Another advantage of patch-wise evaluation is that the patch context focuses on image features that are relevant for defect detection, while ignoring unimportant ones. As with other supervised methods that use oversampling techniques, attention must be given to possible overfitting. However, this method is easily scalable depending on data availability and thus can be fine-tuned as new defects emerge throughout the production process.

Above tasks can be seen as an applicable data-preparation as well as preprocessing workflow in industrial applications, reducing elaborate labeling only to known defective samples and sample regions.

5.2Synthetic defect method

Another method used in this publication is based on the synthesis of artificial defects [48]. This algorithm enables the synthetization of defects with a wide range of appearances, imitating a large proportion of real occurring defects. Basically, the synthetization algorithm consists of four steps:

By utilizing different sets of hyperparameters of the random variables used in steps 1–3, it is possible to generate a variety of different defect morphologies (straight, jagged, curved, circular skeletons, etc.) and characteristics (contrast, intensity distribution). Depending on the appearance of the real defects to be imitated (elongated, punctual as in Fig. 2), the hyperparameters that determine the distribution of the random variables must be chosen selectively.

Due to the transition region of direct reflection as well as the frequent occurring ROI borders, above described method was adapted to generate visually apparent defects in both bright and dark contrasted areas, exclusively within the ROIs. Thus, defect synthetization categories and their underlying hyperparameters were adjusted based on real defects to produce bright and dark contrasted punctate and filamentous morphologies of different sizes and characteristics as depicted in Fig. 7. The preprocessing to generate the training and validation dataset follows almost the same procedure as steps 1–5 in Section 5.1. Additionally, following the central cropping in step 4, defects are generated in 50% of the fault-free patches. As a result, the generated training and validation datasets are balanced. With this method, it is therefore possible to perform balanced supervised training without the need for defective data. However, the generalization ability is only assessable using a test dataset containing real defects. Since the synthetization algorithm is based on grayscale, RGB images, if present, must either be converted or their channels processed independently.

Figure 7.

Synthetic generated defects on fault-free patches of LSM-1 (top row) and ASM (bottom row). Hyperparameters were chosen to mimic punctual and elongated “real” defects as shown in Fig. 2.

5.3Thresholding algorithm

LSM-2 enables the detection of pinholes, pattern misalignments or general inhomogeneities with low optical density. Due to the characteristics of the transmission measurement, defect features typically appear as white dots or areas of certain dimensions (Fig. 2, 2A/2C) within the image patch. To detect these features, a thresholding algorithm was developed, which is briefly described in the pseudo code of Algorithm 5.5.

The connected_components() function groups connected regions and assigns labels to the binarized image patches. These labels are then used to calculate the area of each defective region by counting the number of connected pixels per label. In order to utilize GPU-level parallelization the pseudo code shown in Algorithm 5.5 was implemented in a batchwise manner. With the help of this algorithm it is possible to tune parameters like RGB-thresholds and min. / max. defect areas, depending on the given quality requirements. As with other traditional image processing methods, no training data is demanded.

5.4Baseline method

The representation-based method by Roth et.al [32], namely PatchCore, is based on the extraction of mid-level features of fault-free patches using a pre-trained DCNN. During the training phase, subsamples of these locally aware patch features are stored in a memory bank. During inference, these features are compared to the extracted features of the image using a nearest neighbor search, resulting in anomaly scores. This method achieves SOTA anomaly detection performance on the MVTec dataset, resulting in an image-levelAUROC of up to 99.6%.

5.5Dataset and method settings

In order to evaluate methods described in section 5, image data were acquired by means of all three measurement chambers (LSM-1, LSM-2, ASM) of the inspection system demonstrator. Pre-sorted fault-free samples as well as defective samples of different defect classes were captured. The samples taken are in a state in which the printing process, including drying, has already been completed. Therefore, defects that occurred during printing are treated as fixed at this stage and no further significant changes are expected. However, the samples originate from different production batches and therefore have a desirable permitted design variability.

A training, validation and test dataset was created for each measurement modality. Each dataset consists of patches cropped from the respective ROIs of the acquired samples, resulting in an evaluation patch size of 256 × 256 px. In order to avoid unwanted correlations between the patches, the extraction of defective and fault-free patches was done on separate samples. For the same reason, fault-free and defective patches were assigned to training, validation and test data from separate samples. This procedure allows for the creation of uncorrelated datasets regarding defect classes and data splits. The training data regarding the supervised oversampling and supervised synthetic defect training were created as described in section 5.1, by extracting 100 000 fault-free patches at random positions within the ROIs. A total of 129 defective patches were collected for the supervised oversampling training. The extraction of overlapping fault-free patches for the creation of the training dataset for PatchCore, as well as for the creation of the validation and test datasets, was accomplished by using a sliding window with a stride of 160 px. The LSM-1 dataset consists of 2014 and 1498 fault-free patches together with 74 and 95 defective patches for validation and testing, respectively. LSM-2 contains 406 and 393 fault-free as well as 44 and 110 defective patches for validation and testing. The validation of the ASM was performed using a balanced synthetic dataset as described in section 5.2, containing 10 000 fault-free as well as 10 000 synthesized defect patches. However, the test dataset contains 2060 fault-free and 74 “real” defective patches. In order to imitate the slice wise image capture process in ASM, the same defects may appear in different illumination areas. In general, the datasets contain common defect classes with respect to their optical modality, which can be divided into point defects such as dust and fiber inclusions, mechanical damages, print defects or pinholes, as well as area defects such as pattern misalignment, punctual grid defects or squeegee strokes, etc. (see Fig. 2). Furthermore, all different defect classes are combined into one main defect class per modality. As shown by the distribution of fault-free and defective patches, imbalanced validation and test datasets were created to mimic typical imbalanced inspection data distributions.

0 -1

– Image Batch (batch_size,channels,height,width)

– Threshold (r,g,b)

– Maximum pixel sum above threshold

– Minimum and maximum feature areas

– Filter kernel size (k)

– Image margin indices

– Predictions if patches are defective or fault-free

– Areas of connected components

patch in batch c in channels i in {1,2,…,height}j in {1,2,…,width}i AND j in margins patch[c][i][j] ←0  continue  patch[c][i][j] ⩽ threshold[c] patch[c][i][j] ←0  patch[c][i][j] ←1

patch[r] AND patch[g] AND patch[b] =1 binary_patch ←1  binary_patch ←0

dilated_patch ← dilation(binary_patch, k)

0< sum(dilated_patch) ⩽px_sum_limcomponents ← connected_components(dilated_patch)  c in components areas ← component_areas(c)  𝐚𝐧𝐲 area_lim_lower⩽areas⩽area_lim_upper predictions ← “defective”  predictions ← “fault-free”

predictions ← “fault-free”

predictions, areas  Thresholding Algorithm

6.1Patch-wise defect detection performance

The following section provides a detailed analysis of the method’s patch-wise defect detection performance, considering typical imbalanced inspection data distributions. The best performing method of each optical modality are highlighted in bold (Table 1). As in LSM-1, the supervised oversampling method achieved an MCC of 0.88 at an FPR of 0.2%, closely followed by synthetic defect training with an MCC of 0.85 and the same FPR. PatchCore resulted in a 6 times higher FPR and an MCC of 0.58. With the ASM setup, the best results were achieved through synthetic defect training with an MCC of 0.74 and an FPR of 0.5%. PatchCore performed even less well than in LSM-1, with an MCC of 0.26 and ten times higher FPR of 5.1%. In the absence of a sufficient number of defective samples, the supervised oversampling in ASM was skipped. Due to the clearly pronounced features in LSM-2, PatchCore performed robustly with an MCC of up to 0.95 and a comparatively low FPR of 0.8%. The thresholding algorithm presented in Algorithm 5.5 resulted in an MCC of 0.99 with only one overlooked defect.

Figure 8.

TP, FN and FP (columns) patch inference results of LSM-1, LSM-2 and ASM (rows) using PatchCore. Large area defects, such as the grid defect in 1A as well as the pattern misalignment in 2A produced clearly pronounced areas of anomaly. However, small and weakly contrasted point defects, such as the print defect and the pinhole (1B/2B), were overlooked. Feature variations in the vicinity of masked border regions (2C/3C) and transition regions of direct reflexion (3A) introduced many false positives.

Figure 8 shows the inference results by means of the PatchCore method including anomaly overlay. Rows 1–3 follow the optical modalities LSM-1, LSM-2 and ASM. Columns A–C are arranged according to TP, FN and FP classification. The image margins ignored during inference are clearly visible in 1A–2C. Large area defects, such as the grid defect in 1A, produced clearly pronounced areas of anomaly, whereas point defects, such as the print defect in 1B, were overlooked. The high allowable variance of the structured pattern, as shown in 1C, led to a large amount of false positive patches. Large distinct defects, such as the pattern misalignment in 2A and high contrasted pinholes, produced large deviating feature vectors with respect to the learned feature embedding, leading to proper defect detection. However, small and low contrasted pinholes as illustrated in 2B were more likely to be missed. As with LSM-1, small feature variations in the vicinity of masked border regions also led to anomaly scores, resulting in false positives as in 2C. The ASM dataset poses a challenge regarding defect detection due to its high contrast variance in the transition region of direct reflection and its frequently appearing masked ROI regions. As shown in patches 3A and 3C, many false positives occurred at the border areas of the ROI as well as in before mentioned transition regions. Possible imperfect ROI segmentation also introduced additional feature variance, leading to false positives. Frequently, TP patches were classified as defective because of these detected regions, whereas large area defects such as squeegee strokes in 3B were partially or not detected at all.

Table 2 illustrates the performance metrics obtained by means of the oversampling and synthetic defect methods in LSM-1 and ASM. For this purpose, the metrics previously shown in Table 1 were divided into their individual defect groups, point and area. In case of the supervised oversampling method in LSM-1, no area defect was overlooked, resulting in a recall of 100%. By utilizing synthetically generated defects as visualized in Fig. 7, a recall of 86% was obtained. Overall, area defects have not been synthesized and their morphology is completely different with respect to their optical modality in LSM-1 and ASM. This can be observed in Fig. 8, comparing the grid defect in 1A with the squeegee stroke defect in 3B. The grid defect contains punctate features, comparable to some synthesized point defects in LSM-1, thus potentially led to a robust recall of the above described 86%. The synthetic defect method performed slightly better than the supervised oversampling method in detecting point defects, resulting in a recall of 65%. In general, the detection of small sized defects in this modality is challenging, due to their weak appearance in surrounded structured patterns (Fig. 2 1A).

However, in the case of ASM, the detection of point defects was superior to LSM-1 with a recall of 88%. Although the morphology of the synthesized defects in ASM is completely different from that of area defects (as shown in Fig. 2, 3C), this defect group achieved a recall of 42%. This can be seen as the ability to learn the underlying distribution of the fault-free data using the influence of the vast amounts of augmented fault-free and synthetic defect patches.

Furthermore, the low FPR of 0.2–0.5% observed with these supervised learning procedures, with respect to the remaining high recall of certain defect groups, indicates strong generalization ability. Another indicator of its robustness is that possible imperfect ROI segmentations, such as those seen in ASM (e.g. holes within darkly contrasted ROI regions), which lead to false positives in PatchCore, are not noticeable during inference utilizing the synthetic defect method.

As shown in Table 1, the supervised oversampling method achieved an MCC of 0.81 and an FPR of 0.3% by leveraging a reduced set of 42 labeled defects. This gives an indication of the scalability of this method when compared to the metrics obtained using 129 labeled defects. As stated above 170–300 patches per ROI are evaluated, thus to avoid any false alarms FPR less than 0.59 resp. 0.33% are required to be applicable for inspection runs. Despite PatchCore, struggling with high FPR, introduced methods are capable of achieving even lower FPR.

6.2Inspection speed

In addition to the evaluation of defect detection performance, the inference speed of the applied defect detection methods was determined. As shown in Table 3, the inference times and their reciprocal, the throughput of patches per second, were measured in combination with their CUDA memory footprint. According to the image processing pipeline as described in Section 4.2, image batches with batch sizes of 128 were chosen for evaluation. However, in the case of PatchCore, the batch size was reduced to one. The use of large feature embeddings of 1.5 and 3.1 GiB resulted in a high computational GPU utilization of up to 91%, preventing any noticeable speed-up by means of larger batch sizes. The test image batches consisted of either dummy data sampled from a uniform distribution, or selected defective and fault-free patches yielding to torch tensors (float32) allocated on the GPU.

The performance measures shown in Table 3 were determined using the arithmetic mean of 5 cycles with 100 repetitions, including preprocessing of the respective methods. Prior to each measurement cycles, a GPU warm-up of 10 repetitions was performed. To account for asynchronous CUDA data processing, inference times were measured using PyTorch’s synchronized CUDA events [88]. The GPU memory footprint of each method was determined using torch.cuda.mem_get_info(), deducting PyTorch’s CUDA context of approximately 1.3 GiB.

As expected, the ResNet18 architecture, used by means of the supervised oversampling and the synthetic defect method, achieved the lowest inference time of 0.18 ms, resulting in a throughput of 5556 patches/s. Inference times larger than 1000 patches/s have already been reported for ResNet architectures in [89]. The dependence of the threshold algorithm on the input data is clearly visible in Table 3. The highest inference time was measured with 1.81 ms in the case of the dummy data and 0.58 ms for fault-free data, missing defective pixels and thus skipping the computation of defective areas. These values represent the estimated upper and lower bounds. Thus, in an industrial setting with a majority of fault-free patches per batch, throughputs of up to 1724 patches/s can be achieved. Due to the exhaustive nearest neighbor search in large feature embeddings, PatchCore led to by far the highest inference times of 32.95 and 118.89 ms. This means that both the thresholding algorithm and the methods that utilize the ResNet18 are approx. 57–183 times faster compared to the PatchCore configuration, that ignores image margins. Although, ignoring image margins for feature embedding and subsequent feature comparison led to a approx. 3.6 times reduction in inference time. For comparison, a thorough evaluation of SOTA anomaly detection methods inference speeds, including PatchCore, can be explored on p. 21 in [35]. The proposed method, namely EfficientAD, achieved the highest throughput of up to 614 patches/s, including CPU to GPU transfers. Thus, in inspection settings described above, embedding-based methods like PatchCore would lead to bottleneck the overall inspection process, not achieving required cycle times. In terms of memory footprint, the thresholding algorithm exhibits the smallest value with 1.2 GiB, followed by ResNet18 with 2.3 Gib and PatchCore with up to 6.0 GiB.

6.3Inspection system demonstrator performance

The inference times determined above do not include all data processing steps during defect inspection as described in Section 4.2, such as registration, ROI segmentation or patch extraction, etc. To account for these computationally intensive tasks and to estimate the overall inspection time of the implemented inspection system demonstrator, separate runs of the samples through all measurement chambers were performed. The best performing methods regarding defect detection performance (bold in Table 1) were therefore chosen for each modality. The time measurements started at the onset of image acquisition in LSM-1 and ended after finishing the post-processing of all modalities to ensure the availability of patch-wise inference results. A total of 10 inspection runs with various samples resulted in an average inspection time of 20.25 ± 0.59 s, thus achieving the required cycle time range of 15–30 s. It has to be mentioned that the estimated overall inspection time is generally dependent on the product size and the resulting amount of evaluated patches as well as method settings. As with other hardware-dependent measurements, speed up leveraging more performant e.g. GPUs can be expected.

In terms of defect detection performance, it was qualitatively observed that the used defect detection methods are capable of detecting the wide range of different defect classes with high sensitivity at low false positive rates, confirming the metrics in Table 1. In the case of LSM-1, small defects that differ only slightly from their surroundings are more likely to be overlooked than area defects that are pronounced over the entire ROI. Despite the observation of missed defects (mostly area defects) in the test dataset of ASM resulting in a relatively low MCC of 0.74 compared to other modalities, this was not the case when performing inspection. Due to the high frame rates in ASM, the possibility of a defect occurring in more than one patch as well as in different illumination regions is increased and thus counteracts the determined dataset-specific recall. However, a thorough quantitative evaluation of the system’s defect detection performance, considering the quality standards of the project partner, should be carried out in the future.