Plane-wave medical image reconstruction based on dynamic Criss-Cross attention and multi-scale convolution

2.1Multi-scale convolution module

Inspired by the multidimensional feature extraction in the literature [11], this paper designs a multiscale module to capture the scattering features in each dimension through convolutional operations at different scales to alleviate scattering interference on image reconstruction. The structure of the multiscale convolution module is depicted in Fig. 2, and Table 1 presents the main parameters of each layer.

Figure 1.

Schematic diagram of the overall structure of the modeU-Net Generator.

Figure 2.

Schematic diagram of multi-scale convolution module structure.

2.2Dynamic criss-cross attention

In image reconstruction, While the widely used self-attention mechanism comprehensively captures an image’s global structure, it fails to extract local image features. Referring to the enhancement of local features in reference [12], this study proposes the DCCA mechanism, which introduces a sliding window on top of Criss-Cross Self-Attention, and the introduction of the sliding window not only enhances the local features but also avoids the occlusion region in the single-beam plane-wave image from interfering with the reconstruction of the target region. DCCA establishes the connection between each pixel point in the sliding window by performing two cross operations inside the sliding window. By adjusting the sliding window’s size and step length, the proportion of global and local features extracted by DCCA can be fine-tuned. The larger the sliding window step size and the smaller the size, the higher the weight of local feature extraction, and vice versa, the smaller the step size and the larger the size, the higher the weight of global feature extraction. The implementation process of DCCA is depicted in Fig. 3.

Figure 3.

Schematic diagram of the Dynamic Criss-Cross Attention structure.

In the first step, a weight matrix is obtained. Three 1×1 convolution operations are performed on each sliding window region xn∈Rh×w×c of the feature map X, where h, w, and c represent the length, width, and number of channels, respectively. These operations generate Qn, Kn, and Vn, where {Qn,Kn}∈Rh×w×c′, Vn∈Rh×w×c, and c′ denotes the number of channels. From Qn, the feature vector Qn⁢μ∈Rc′ is extracted at position μ. By selecting all vectors from Kn centered at point μ in the horizontal and vertical directions, a set Kn⁢μ∈R(h+w-1)×c′ is obtained, multiply the vector Qn⁢μ with the set Kn⁢μ, and perform this operation for each pixel point in Qn in turn to obtain Fn∈Rh×w×(h+w-1). Applying softmax normalization on Fn along the channel direction produces a weight matrix An⁢1 that encapsulates the relationships between pixel points in the cross direction, as illustrated by the gray arrows in the figure.

In the second step, weights are applied to the input feature map. The weight vector An⁢1⁢μ∈Rh+w-1 in the channel dimension is obtained at the position μ of the weight matrix An⁢1. All pixel points on any of the channels from Vn are selected with the point μ as the center in the horizontal and vertical directions to form the vector Vn⁢i⁢μ∈Rh+w-1, where i(i=1,2,…,c) denotes the channel number in Vn. The vector In⁢μ∈Rc is obtained by multiplying An⁢1⁢μ with the vector Vn⁢i⁢μ on each channel in Vn in turn, and the vector In⁢μ obtained is added with the vector xn⁢μ∈Rc of xn at μ to obtain the vector zn⁢μ∈Rc with the information of pixel points in the cross direction. This process is performed for each pixel point in xn, thereby introducing cross-directional information into each pixel point within the sliding window region, as depicted by the black arrow in the figure.

In the third step, the first and second steps are repeated. Ultimately, the feature region zn′ is obtained, establishing the association of any pixel point with the rest of the pixel points in the sliding window region.

The deep layer of the U-Net decoder utilizes DCCA with a relatively small sliding window size to focus on enhancing texture information. In contrast, the shallow layer employs DCCA with a relatively sizeable sliding window size to focus on the fusion of general information. The specific parameters of DCCA in the U-Net decoder are shown in Table 2.

2.3Loss function

This study introduces L1-SSIM loss and perceptual loss on top of the original adversarial loss. The combination of L1 loss and MS-SSIM loss (L1-SSIM loss) considers the generated image’s gray value and structural information. The perceptual loss function encourages the generated and actual images to contain similar high-level semantic information.

3.1Experimental details

For this study, we utilized the dataset from the 2016 Plane-Wave Imaging Challenge in Medical Ultrasound (PICMUS), comprising 360 image sets. The dataset included 120 sets of simulated images generated using Field II, 120 sets of experimental images obtained through the Verasonics Vantage 256 study scanner and L11 probe, and 120 sets of human carotid images. Image flipping operations were performed to augment the dataset and enhance its diversity, resulting in 1350 image sets. Subsequently, the dataset was divided into training and test sets in an 8:2 ratio.

Different evaluation measures were selected to assess the reconstruction performance for each type of image. The Full Width at Half Maximum (FWHM) and Contrast Ratio (CR) were utilized to evaluate the reconstruction of point and cyst target images, respectively. Furthermore, the PSNR and SSIM were employed to evaluate the reconstruction of actual images in vivo.

Full Width at Half Maximum: The FWHM indicates the beamwidth of the main lobe at -6 dB on the point spread function.

Contrast Ratio: The CR quantifies the level of differentiation between the target area and the background area.

Peak Signal-to-Noise Ratio: The PSNR measures the global pixel error between the generated and actual images to assess the degree of image reconstruction.

Structural Similarity Index: The SSIM is an index based on human visual perception, measuring the similarity between the generated and actual images.

3.2Comparative experiments

3.2.3Model scale comparison

As shown in Table 5, MD-GAN has a certain advantage in terms of the model scale, and compared with the SRGAN model, it is less computationally intensive and more time efficient despite the more significant number of parameters.

Table 5

A comparison of model scale

Method	Parameters	MACs(G)
HRGAN [13]	873.4534K	172.1326
SRGAN [14]	839.6215K	165.1523
DCGAN [15]	35.1916G	72.3446
ResGAN [16]	31.4203G	142.7519
AUGAN [17]	43.8314G	53.8906
UNetGAN [18]	42.6963M	53.2867
Our method	43.6180M	63.1105

Figure 4.

Comparison plots of experimental results of deep learning methods.

3.3Ablation experiments

3.3.1Ablation experiment of the generator part

The effectiveness of the U-Net network improvement in this model was evaluated through ablation experiments in the generator section, as presented in Fig. 5. The scatter in the generated images is significantly reduced by incorporating the multi-scale convolution module. Moreover, the addition of DCCA resulted in significantly enhanced simulated point targets. The color of the obtained cyst target was also deepened by adding DCCA to the reconstruction of the cyst target. Furthermore, in reconstructing the actual image in vivo, the addition of DCCA was able to reconstruct the carotid artery image with a more complete overall structure.

Table 6

Results of generator ablation experiments

Generator	FWHM (mm)	CR(dB)	In vivo target
	(sim/exp)	(sim/exp)	PSNR(dB)	SSIM
DCCA	0.2883/0.4871	35.0633/25.2113	30.1439	0.9113
Mult	0.2936/0.4896	29.4425/23.4765	32.1721	0.9362
DCCA+mult	0.2844/0.4822	36.8092/25.9520	33.5419	0.9577

Figure 5.

Plot of the ablation experiment’s results for the generator part.

As presented in Table 6, In the case of point targets, incorporating DCCA resulted in a reduction of 0.0092 mm in the FWHM of simulated point targets. Cyst targets, including DCCA, yielded a significant enhancement in CR for both simulated and experimental cyst targets, with improvements of 25.02% and 10.54%, respectively. In the reconstruction of real in vivo images, utilizing the multi-scale convolution module led to an increase in PSNR of 3.3980 dB and an improvement in SSIM of 0.0464 for the carotid reconstruction images.

3.3.3Ablation experiments of sliding windows in DCCA

In this study, ablation experiments were conducted to investigate the impact of sliding windows of DCCA in the U-Net decoder. Specifically, as depicted in Fig. 7, the reconstructed simulated point targets appeared noticeably weaker when the 7 × 7, 5 × 5, and 3 × 3 sliding windows were removed. Furthermore, removing the sliding windows of 9 × 9, 5 × 5, and 3 × 3 led to overall darker reconstructed experimental and carotid artery images.

Table 8

Results of the DCCA’s sliding window ablation experiments

Sliding window	FWHM (mm)	CR(dB)	In vivo target
	(sim/exp)	(sim/exp)	PSNR(dB)	SSIM
3*3+5*5+7*7	0.2847/0.4836	34.6919/25.7749	31.5622	0.9561
3*3+5*5+9*9	0.2864/0.5061	34.4011/25.7141	32.2631	0.9447
3*3+7*7+9*9	0.2891/0.4885	36.1980/25.9410	29.5940	0.9170
5*5+7*7+9*9	0.2937/0.4993	34.9729/25.8108	32.4483	0.9444
3*3+5*5+7*7 +9*9	0.2844/0.4822	36.8092/25.9520	33.5419	0.9577

Figure 6.

Plot of the ablation experiment’s results for the loss function part.

Figure 7.

Plots of experimental results for ablation of sliding windows in DCCA.

Table 8 presents the effects of sliding windows in DCCA on various target reconstructions. For point target reconstruction, including a 3 × 3 sliding window, it resulted in a 3.17% reduction in the FWHM of the simulated point target. Adding a 7 × 7 sliding window led to a 4.72% decrease in the FWHM of the experimental point target. In the case of cyst target reconstruction, introducing a 7 × 7 sliding window increased the CR of the simulated and experimental cyst targets by 7.00% and 0.93%, respectively. Moving on to the reconstruction of in vivo targets, incorporating a 5 × 5 sliding window improved the PSNR and SSIM in carotid images by 3.9479 dB and 0.0407, respectively.

4.1Contribution of modules

In medical image reconstruction, the use of self-attention, although better at establishing the overall structure of the image, can be influenced by the occluded areas of the single-beam plane-wave image in the process of extracting global features, resulting in the overall darkness of the reconstructed experimental and in vivo images. The DCCA proposed in this paper effectively solves the above problems. The introduction of the sliding window in DCCA can avoid the interference of irrelevant regions to the target region in the reconstruction process, increase the local feature extraction ability while retaining the traditional self-attention global feature extraction ability, enhance the cyst target and simulation point target while ensuring the reconstruction effect of the background region, improve the CR of the cyst target image, and reduce the FWHM of the simulation point target image.

The addition of a multi-scale convolution module to the encoder of the U-Net network aids in separating the high and low-frequency information from the input image through the use of convolutional kernels of varying sizes. This separation facilitates the determination of the boundary of the simulated point target and eliminates scatter around the simulated point target. The multi-scale convolution module, which combines DCCA in the decoder, can more effectively utilize the separated high and low-frequency information, reducing the FWHM of the point target image and an improvement in the PSNR of the carotid artery image.

In this study, the loss function of the MD-GAN model introduces L1-SSIM loss and perceptual loss on top of the original adversarial loss function. The L1-SSIM loss can guarantee the grayscale value of the generated image while building a clear image structure, ensuring that the generated image maintains high structural similarity with the actual image. Additionally, the perceptual loss encourages the generated image to contain similar high-level semantic information as the actual image. The original adversarial loss alone is insufficient to establish the actual structure of the image. However, the combination of adversarial loss, L1-SSIM loss, and perceptual loss leads the model to converge in the optimal direction, resulting in the generation of plane-wave ultrasound images with complete structure and precise details.

4.2Model quality

The MD-GAN model exhibits excellent performance in point target images, cyst target images, and in vivo carotid artery images, and it surpasses several other evaluation models in terms of visual effects and various evaluation metrics. However, it is essential to note that deep learning-based reconstruction models achieve image-to-image reconstruction by learning the nonlinear relationship between low-resolution and high-resolution images without fully utilizing beam information. Consequently, the FWHM of the generated experimental point target images and the PSNR of carotid artery images still do not outperform the performance of beam-based reconstruction methods.

Deep learning-based methods demonstrate substantial fluctuations in overall reconstruction results in point target images. Specifically, SRGAN yields noticeably poor reconstruction results. DCGAN and HRGAN models generate the point target image with significant scatter and unsatisfactory performance across all experimental indexes. While ResGAN manages to reduce the FWHM of both simulated and experimental point target images, evident reconstruction traces are present in the occluded areas of the images. The UNetGAN model exhibits a limited ability to extract low-frequency information and fails to accurately reconstruct the background area of the experimental point target image. Introducing an attention mechanism in AUGAN enhances the model’s ability to extract high-frequency features and performs better in reconstructing simulated point target images. However, the model has limited capability for extracting global features, resulting in poor reconstruction quality in the masked region of the experimental point target image. By integrating DCCA with a relatively sizeable sliding window size into the shallow layer of the decoder, MD-GAN effectively enhances the low-frequency information extracted from the multi-scale convolution module, thereby achieving high-quality reconstruction of occluded and background regions. The utilization of the multi-scale convolution module separates the high and low-frequency information, resulting in accurate reconstruction of point target boundaries and effectively eliminating scattered spots surrounding the point target. Furthermore, incorporating DCCA with a relatively small sliding window size in the deep layer of the decoder enhances local features, resulting in a more prominent point target. Consequently, the model successfully reduces the FWHM of the generated image while producing reconstructed images with clear targets and complete structures.

Regarding reconstructing cyst target images, the SRGAN model yields terrible visual effects and image metrics. DCGAN and HRGAN generate images with low CR, and there are severe noise and image blurring problems, respectively. While ResGAN manages to improve the CR of cyst images, it fails to recover obscured parts of the images. The introduction of attention in the AUGAN model significantly deepens the cyst target in the reconstructed image and improves the CR of the simulated cyst target. However, the model exhibits a poor ability to acquire low-frequency information, resulting in overall darkness in the generated cyst images and slight reconstruction traces. The UNetGAN model struggles to accurately extract global features of the image, leading to too bright and too dark areas in the background of the generated image. Although high CR is achieved in reconstructing the experimental cyst target, the model needs to improve the quality of the generated images. The MD-GAN model adds DCCA with increasing sliding window size sequentially from deep to shallow layers of the U-Net decoder. This approach increases the weight of local feature extraction before increasing the weight of global information extraction, ensuring that the generated image possesses a more precise overall structure while highlighting the cyst target and improving the CR of the cyst target image.

The MD-GAN model outperforms the other evaluated models in reconstructing in vivo carotid artery images, primarily due to the rest of other evaluation models’ inability to effectively extract low-frequency features and accurately reconstruct the global structure. MD-GAN combines a multi-scale convolution module with DCCA to capture low-frequency information precisely and introduces the L1-SSIM loss function to positively guide the structural similarity between the generated and actual images. Nevertheless, compared to traditional beam synthesis methods like CF and GCF, the reconstruction effect of MD-GAN in vivo carotid artery images still has room for improvement.