Subcutaneous fat predicts bone metastasis in breast cancer: A novel multimodality-based deep learning model

1.Introduction

Breast cancer (BC) is a prevalent cancer that is most commonly found among female cancers and is the second leading cause of cancer death among women globally [1]. Bone metastasis is one of the most common metastatic sites in BC patients [2]. Patients with bone metastasis are accompanied by osteoclast-mediated bone destruction and have a 5-year overall survival rate of 22.8% [3, 4]. Early detection of bone metastasis is crucial for improving survival. However, the predictors for bone metastasis have not been sufficiently elucidated. Therefore, there is an urgent need for identifying novel markers for bone metastasis.

Obesity is associated with an increased risk of death in the general population [5], but this contradicts with some reports on the relationship between obesity and mortality in cancer patients [6, 7]. An important reason for the ‘obesity paradox’ may be that body mass index (BMI) cannot distinguish well between individual muscle and fat tissue distributions [8, 9]. Additionally, a higher BMI can mask low muscle mass, and a lower BMI can mask excess obesity. Subcutaneous adipose tissue (SAT) and visceral adipose tissue (VAT) are two different physical forms of white adipose tissue [10]. The VAT is currently thought of as a metabolic and endocrine organ that can influence systemic immunological disorders and body weight homeostasis. SAT, a metabolic storehouse, is linked to visceral fat deposition [11]. In a number of tumor types, including breast cancer, colorectal cancer, hepatocellular carcinoma, and gastric cancer, elevated VAT has been associated with a worse overall survival (OS) rate [12, 13, 14]. Similarly, the SAT index can be used to predict the outcomes of several cancers, such as head and neck, breast, and prostate cancer [15, 16, 17]. Visceral to subcutaneous fat area ratio (VSR) is an independent prognostic factor for poor prognosis in type 1 endometrial cancer and gastric cancer [18, 19]. Recent studies have demonstrated that adipocyte-BC cell interactions are critical for the development of BC and its related bone metastases [20].

Radiomics involves extracting quantitative characteristics from digital images and converting the data into high-dimensional information. Its primary objective is to develop decision-support tools [21, 22]. Typical radiomics includes image segmentation within the region of interest (ROI), followed by the selection and extraction of features such as size, shape, and texture. Statistical methods or machine learning techniques are then utilized to derive the final clinical outcomes [23]. Deep learning (DL) is widely applied in medical image analysis due to its exceptional performance [24]. DL algorithms can extract features from medical images that are beyond human recognition capabilities, enabling automatic quantitative evaluations without introducing additional errors associated with manual feature extraction. However, many studies only employ a single set of medical images, lacking comprehensive clinical information, thus limiting the final predictive outcomes. Multimodal deep learning models offer the potential to encompass additional modalities beyond image data [25, 26, 27].

Local binary pattern (LBP) serves as an efficient texture description technique. By comparing the gray value of each pixel with its neighboring pixels, it characterizes the image’s texture features using a two-level system [28, 29, 30]. This method finds extensive use in face recognition, yielding favorable results [31, 32, 33]. Histograms of oriented gradient (HOG), an algorithm for extracting feature histograms from local pixel blocks [34], have enjoyed significant success in object detection, particularly within pedestrian detection scenarios [35, 36, 37]. LBP boasts advantages in rotation and grayscale invariance, effectively capturing image texture features, while HOG excels in capturing local shape information, maintaining strong invariance to geometric and optical variations. Consequently, convolutional neural networks (CNN) can extract a broader spectrum of features from LBP and HOG features, which diverge from the original computed tomography (CT) images.

In this study, we have developed and validated a neural network-based DL algorithm to assess predictors for bone metastases in breast cancer patients. Additionally, regularized Cox regression models were employed to analyze independent prognostic factors for bone metastases.

2.Materials and methods

2.1Patients and data sets

This study protocol was approved by the Ethics Committee of Harbin Medical University Cancer Hospital. As it was a retrospective study, informed consent from all participants was exempted. The study comprised 431 patients who underwent radical surgical resection at Harbin Medical University Cancer Hospital between January 1, 2015, and December 31, 2016. The inclusion criteria for this study were as follows: (1) Patients did not receive neoadjuvant chemotherapy or other treatments before surgery. (2) The patients’ age was at least 18 years. (3) Pathological findings were histologically confirmed. (4) Patients had complete clinical and follow-up data. (5) Patients had no history of other malignancies and no metastatic disease at the time of diagnosis. The patients were assigned to the training cohort, validation cohort, and test cohort.

All clinical information, including age, menopausal status, histology type, and body composition, was retrieved from the records of breast cancer (BC) patients. Distant metastasis-free survival (DMFS) was computed from the surgery date to the occurrence of distant metastasis or the last follow-up date, with assessments conducted every 3 months during the first 2 years post-operation and subsequently every 6 months for the 3–5 years thereafter. Distant metastasis outcomes were derived from patient imaging conducted during the follow-up period. Body mass index (BMI, kg/m²) is calculated by dividing weight (kg) by height squared (m²). The erector spinae area, visceral adipose tissue area, and subcutaneous adipose tissue area were obtained with Image J software version 1.53a (Wayne Rasband National Institutes of Health, USA). Different tissues were differentiated based on CT Hounsfield Units (HU). HU was set from -29 to +150 for skeletal muscle, -150 to -50 for visceral adipose tissue, and -190 to -30 for subcutaneous adipose tissue. Area (cm²⁾ divided by height (m²) squared to obtain erector spinae index (ESMI), visceral fat index (VFI), and subcutaneous fat index (SFI), as shown in Fig. 1. Because there were no CT images available on level L3 for BC patients, CT images on the level of the eleventh thoracic vertebra (T11) were used as an alternative for the assessment of muscle and fat. Two independent readers blinded to the clinical data analyzed the CT images, and the mean of the two measurements was used.

Figure 1.

CT image of T11 level, paraspinal muscles, subcutaneous fat and visceral fat. Figure (a) is the CT image of the T11 level. The yellow area in Figure (b) is the visceral fat in the T11 level CT image. The yellow area in (c) is the subcutaneous fat in the T11 level CT image. The yellow area in (d) is the paraspinal muscles in the T11 level CT image.

2.3Feature Extraction with HOG and LBP

Figure 2.

LBP feature and HOG feature. Figure (a) is the LBP feature extracted from the T11 layer CT image. Figure (b) is the HOG feature extracted from the T11 layer CT image.

The HOG and LBP features were respectively extracted from the CT images, as illustrated in Fig. 2. For HOG, features were calculated using a unit size of 16 × 16 and a block size of 1 × 1, with 8 bins for the histogram. As for LBP, the range radius parameter was set to 3, and the domain contained 24 pixels. HOG operates on the local grid units of the image, ensuring robust geometric and deformation invariance. Meanwhile, LBP features offer notable advantages, including grayscale and rotation invariance. These two features were incorporated into the CNN for training and subsequent feature extraction.

2.4Deep learning model training and interpretation

Figure 3.

The overall pipeline of the model. The fat area mask, LBP feature, HOG feature, and CT image are combined according to the channel dimension to form the input. ResNet combines the attention mechanism guided by the fat mask to form an encoder model to extract features from image data. Lasso-Cox screens clinical information data for features, and then combines them with extracted network features to form multimodal data. GBDT is a classifier for multimodal data.

The study’s workflow is depicted in Fig. 3. To bolster model robustness and mitigate overfitting concerns, we implemented horizontal and vertical flipping, alongside standardized data augmentation techniques [38]. For a more focused emphasis on the subcutaneous fat area, we incorporated a mask-guided attention mechanism, amplifying the mask area’s responsiveness. The mask-guided attention process is detailed in the appendix. All images were standardized to a size of 500*400 pixels, ensuring uniform distance scaling. Preprocessing was facilitated by the Torchvision toolkit (version 0.11.3) within Python (version 3.8.12). PyTorch 1.10.2 was employed as the backend for all model training.

The 431 patients were equally divided into three groups for three-fold cross-validation, with two of them being used as the training cohort and the other as the test cohort. Then, 20% of the training cohort was randomly selected as the verification cohort. The training cohort was employed for updating the CNN model weights, while the validation cohort assisted in guiding the selection of model hyperparameters. The network model’s weights were initialized using the pre-trained model from ImageNet. ResNet101 was chosen as the foundational model for feature extraction.

Before training the CNN, each patient’s image was assigned a label of either 0 or 1, based on the presence or absence of bone metastases. During the training phase, the improved image was fed into the CNN, and the CNN’s parameters were refined through the process of backpropagation. The model employed the CrossEntropyLoss function as its loss metric, while the Adam optimizer updated the model’s parameters using a batch size of 64, a learning rate of 1e-6, and 200 iterations. Further elaboration on the training outcomes can be found in the appendix.

3.Results

3.3Performance of multimodal models

Table 4

The prediction of bone metastasis result

Methods		AUC	ACC (%)	SENS (%)	SPEC (%)
ResNet101 + GBDT (only CT)	T	0.913 [0.855, 0.951]	83.5 [79.8, 87.3]	84.7 [79.8, 89.6]	83.3 [75.6, 86.9]
	V	0.711 [0.588, 0.773]	69.2 [58.1, 80.9]	72.5 [61.9, 84.2]	70.8 [67.2, 83.1]
	I-T	0.700 [0.594, 0.793]	67.1 [60.7, 75.3]	69.2 [61.8, 77.0]	66.8 [60.1, 76.4]
ResNet101 + GBDT (only clinical information)	T	0.935 [0.890, 0.964]	87.2 [81.0, 89.4]	90.9 [83.9, 93.0]	93.4 [90.4, 95.7]
	V	0.829 [0.630, 0.956]	77.2 [69.0, 84.4]	73.7 [62.1, 86.7]	77.6 [66.2, 89.1]
	I-T	0.783 [0.672, 0.878]	74.5 [64.9, 80.9]	75.0 [67.6, 80, 9]	74.5 [67.9, 80.0]
ResNet101 + GBDT (CT and clinical information)	T	0.990 [0.966, 0.998]	96.1 [93.4, 97.8]	96.5 [92.3, 98.6]	96.0 [93.7, 98.1]
	V	0.939 [0.800, 0.982]	90.1 [78.3, 97.7]	89.5 [78.6, 95.3]	90.1 [80.3, 97.8]
	I-T	0.922 [0.843, 0.964]	83.1 [75.2, 89.5]	82.7 [74.1, 90.2]	83.2 [75.0, 91.0]
ResNet101 + SVM (only CT)	T	0.813 [0.746, 0.873]	75.4 [69.8, 79.8]	77.6 [70.2, 81.8]	75.0 [72.6, 84.1]
	V	0.645 [0.558, 0.792]	59.6 [50.2, 69.8]	57.9 [46.2, 68.2]	59.9 [50.2, 71.2]
	I-T	0.681 [0.570, 0.784]	60.6 [54.6, 67.4]	61.5 [54.5, 66.6]	60.5 [53.2, 70.4]
ResNet101 + SVM (only clinical information)	T	0.859 [0.789, 0.917]	79.6 [75.1, 83.6]	77.6 [67.1, 82.2]	79.8 [73.6, 85.1]
	V	0.765 [0.621, 0.891]	71.3 [60.1, 83.2]	73.7 [62.9, 85.0]	71.1 [60.7, 83.5]
	I-T	0.786 [0.672, 0.881]	75.0 [69.7, 80.2]	76.9 [69.5, 83.2]	74.7 [66.3, 79.8]
ResNet101 + SVM (CT and clinical information)	T	0.959 [0.918, 0.983]	88.7 [86.1, 92.0]	90.6 [86.7, 94.2]	88.4 [83.5, 93.1]
	V	0.864 [0.688, 0.964]	77.2 [65.7, 89.1]	73.7 [61.1, 86.3]	77.6 [66.9, 87.6]
	I-T	0.855 [0.760, 0.928]	76.6 [71.2, 84.3]	78.8 [71.3, 84.4]	76.3 [69.7, 85.4]
ResNet101 + RF (only CT)	T	0.980 [0.951, 0.993]	92.6 [90.0, 95.5]	94.1 [89.9, 98.1]	92.4 [86.1, 97.5]
	V	0.612 [0.514, 0.763]	55.6 [48.9, 67.2]	58.8 [49.1, 68.6]	52.9 [44.2, 61.4]
	I-T	0.638 [0.532, 0.742]	61.8 [57.4, 68.0]	63.5 [58.7, 67.2]	61.6 [55.2, 67.6]
ResNet101 + RF (only clinical information)	T	0.991 [0.958, 0.997]	92.8 [89.7, 95.9]	95.3 [90.1, 97.2]	92.4 [85.3, 96.9]
	V	0.835 [0.708, 0.921]	80.8 [69.3, 89.7]	75.2 [64.7, 86.0]	79.8 [66.5, 90.2]
	I-T	0.785 [0.688, 0.865]	74.3 [68.3, 80.4]	73.1 [67.5, 80.3]	74.5 [68.3, 81.4]
ResNet101 + RF (CT and clinical information)	T	0.995 [0.966, 0.997]	93.6 [90.9, 96.7]	96.5 [90.0, 98.4]	93.2 [86.9, 96.2]
	V	0.856 [0.670, 0.948]	76.1 [65.6, 87.2]	79.7 [68.5, 89.7]	74.1 [60.2, 87.6]
	I-T	0.839 [0.748, 0.913]	77.3 [70.0, 86.1]	78.8 [70.6, 87.6]	77.1 [68.9, 85.2]

95% confidence intervals included in brackets. AUC area under the receiver operating characteristic curve, ACC accuracy, SENS sensitivity, SPEC specificity. GBDT gradient boosting decision tree, SVM support vector machine, RF random forest. T training cohort, V validation cohort, I-T independent test cohort.

We explored various machine learning models to integrate clinical information with network features extracted by CNN. The results are presented in Table 4. Notably, the optimal performance was achieved when employing the GBDT model to amalgamate different feature predictions. Combining clinical information with features extracted by ResNet101 (utilizing feature extraction from T11 level CT images, HOG, LBP, and mask), yielded the most favorable predictive outcomes, with an AUC of 0.922 (95% CI: 0.843, 0.964, P< 0.01) based on three-fold cross-validation. In assessing sensitivity and specificity, we experimented with varying cutoff values on both the training and validation sets. Eventually, we selected a threshold that yielded a relatively balanced trade-off and applied it consistently across the training, validation, and test sets. Considering the comprehensive AUC metric, this model exhibited the most superior classification performance. The ROC curves for different models are illustrated in Fig. 4, providing a visual representation of their comparative discriminative abilities.

Figure 4.

Comparison of receiver operating characteristic (ROC) curves between different models for predicting bone metastases. Figure (a) is the ROC image on the test set when different ResNet models only have CT. Figure (b-c) are ROC images of different machine learning models under different data conditions.

Figure 5.

Correlation coefficient heat map between different modal features. The features from X1 to X16 represent the CNN network features extracted from the combination of CT, HOG, LBP, and Mask, while the remaining features encompass clinically informative attributes.

We calculated the Pearson correlation coefficient between data of different modalities (clinical information and network features) and drew it into a heat map, as shown in Fig. 5. It can be seen that in addition to the strong linear relationship between network features (X1–X16), body composition parameters (BMI-VFI) also have a certain linear relationship with network features, which shows that CNN has extracted features that are similar to body composition. In addition, network features are less correlated with other clinical features except body composition parameters, and feature fusion can play a complementary role.

4.Discussion

In this study, we have identified SFI as an independent prognostic factor for bone metastases in breast cancer (BC) patients. Additionally, we have successfully developed a multimodal prediction model for bone metastases in BC patients, employing a combination of deep learning and GBDT techniques. Our multimodal model has demonstrated promising predictive performance on the test cohort, yielding an AUC of 0.922, a sensitivity of 82.7%, and a specificity of 83.2%. These outcomes underscore the viability of employing a multimodal approach in predicting bone metastases in BC patients. Such a model holds potential to enhance diagnostic proficiency, particularly among less-experienced physicians. To our knowledge, this study represents the pioneering utilization of a multimodal model for predicting bone metastases in BC. Table 5 presents a comparative analysis of our results with other pertinent studies focused on predicting breast cancer metastasis. Notably, our current AUC and ACC models exhibit superior performance compared to alternative models utilizing diverse datasets [40, 41, 42, 43, 44, 45, 46].

Figure 6.

Responses of several convolutional layers in ResNet to CT images of different patients. Grad-CAM results of three CT images. Red areas represent a greater impact on the prediction, blue areas the opposite.

Low SFI is independently associated with increased mortality and poorer survival in cancer patients, as indicated by prior studies [47]. Dong et al. similarly discovered that reduced subcutaneous fat was linked to a more adverse prognosis for gastric cancer, encompassing overall survival and disease-free survival [48]. Black et al. also reported that diminished subcutaneous fat has been correlated with reduced survival in operable colorectal cancer [49]. Furthermore, a recent study substantiated that lower SFI independently predicted poor overall survival in hepatocellular carcinoma [50]. Another study proposed that patients with bone metastases exhibiting elevated SFI and VFI demonstrated superior overall survival [51]. However, Bradshaw et al. observed that, among women with non-metastatic breast cancer, increased SAT was associated with shorter survival [16]. These divergent findings might stem from variances in study populations, inclusion criteria, and biomarkers of SAT. Consequently, further research is warranted to elucidate the role of SAT in breast cancer.

Up to the present date, clear identification of potential explanations for the predictive effect of SAT on bone metastases remains elusive. SAT stands out as the primary source of adiponectin and leptin, pivotal players in the regulation of bone health and bone metastasis in BC [52]. Adiponectin exhibits pro-apoptotic and anti-proliferative properties within human BC cells [53, 54]. Research has indicated that adiponectin hampers the metastatic process through its capacity to suppress the adhesion, invasion, and migration of BC cells, facilitated by the activation of the AMPK/S6K axis and the upregulation of live kinase B1 (LKB1) [53]. In sum, compelling evidence suggests that lower levels of circulating adiponectin forecast heightened BC risk and a less favorable prognosis [55]. Moreover, the mass of adipose tissue corresponds directly to leptin synthesis and plasma levels, enhancing lipid metabolism and insulin sensitivity [56]. Leptin, in turn, exerts a regulatory role on bone health by modulating bone density, growth, and adiposity. Investigations have unveiled diminished serum leptin levels in premenopausal BC patients compared to their healthy counterparts [57]. Furthermore, leptin has also shown promise in correlating with improved prognosis among patients with colorectal cancer [58]. Hence, it emerges that adipokines originating from adipose tissue and leptin influence distinct phases of the bone metastatic cascade. Nonetheless, further inquiry is warranted to validate the mechanisms through which SAT potentially safeguards against bone metastasis in BC.

Within our cohort, we have also demonstrated a notable association between CEA, molecular subtype, and LVI in multivariate analysis. In a retrospective study by Chen et al., CEA was not observed to independently predict bone metastasis in BC; however, a significant difference in the distribution of bone metastasis in breast cancer was noted [59]. Further research and validation are essential to determine the significance of CEA in predicting the likelihood of bone metastasis in breast cancer. A SEER population-based study showed that histologic type was an independent factor for bone metastases in breast cancer [60]. The study carried out by Nishimura et al. demonstrated a strong correlation between KI-67 expression and bone metastasis in cases of breast cancer [61]. A separate retrospective study indicated that ER+HER2- tumors exhibiting a Ki67 score exceeding 13% displayed the highest incidence of bone metastasis [62]. Recent work by Floyd et al. has underscored the pivotal role of LVI in prognosticating overall survival and disease-free survival in breast cancer [63]. In summation, the existing body of literature emphasizes the pivotal significance of KI-67 and LVI in anticipating bone metastasis among breast cancer patients, aligning harmoniously with our findings.

Meanwhile, no association between BMI and bone metastasis was observed in our study. The relationship between BMI and postoperative outcomes in cancer patients has been a subject of controversy. A previous report unveiled the inconsistency of the association between BMI and cancer survival across diverse cancer types and stages [56]. Disparities in the prognostic implications of BMI may, in part, arise due to variations in body composition among individuals with comparable BMI values (e.g., more fat or muscle in one patient than in another) [64]. In essence, BMI falls short in discerning between fat and fat-free mass or distinguishing diverse fat deposits for an accurate assessment of body composition [65]. In this study, we opted for regional measurements of obesity, such as SFI and VFI, which mitigate the risk of misestimation and enhance the coherence of our findings.

Figure 7.

Kaplan-Meier analysis of bone metastases stratified by SFI. Patients were grouped with an SFI of 27.7 as the cutoff value. The difference in the probability of bone metastasis between the two groups was compared.

Our research has unearthed the substantial potential of the SFI in stratified prognostication of breast cancer bone metastasis. Drawing from our findings, we have pinpointed a critical threshold of 27.7 cm²/m², yielding a sensitivity of 0.727 (95% confidence interval: 0.694–0.751) and a specificity of 0.743 (95% confidence interval: 0.701–0.785). The Kaplan-Meier curve for bone metastasis stratified by this threshold is shown in Fig. 7. This discovery carries profound clinical implications, capable of empowering medical practitioners to arrive at more precise diagnoses and therapeutic choices. In the context of predicting breast cancer bone metastasis, we propose that an individual’s SFI can be measured to serve as a pivotal risk assessment marker. Subsequently, this marker can be judiciously stratified to align with specific diagnostic requirements. Furthermore, a pertinent avenue to explore involves incorporating the SFI as an adjunctive feature within the artificial intelligence model’s training phase, thus constituting an integral facet of “feature engineering.” This strategic inclusion empowers the model to harness SFI-derived insights, culminating in heightened prediction precision.

Studies have demonstrated that CNN can effectively employ multi-view bone scan images to automatically diagnose bone metastases [66, 67]. For the prediction of lymph node metastasis, CNN has been utilized to extract features from ultrasound images and SWE images [68, 69]. In our study, we showcase the efficacy of a deep learning model based on CNN in conjunction with the GBDT method for the accurate prediction of bone metastases. This is achieved by integrating CT image features with the clinical information of BC patients. Additionally, we establish that the inclusion of HOG and LBP features alongside CT image features leads to further enhancement of predictive performance. In contrast to traditional qualitative reasoning, quantitative evaluation of imaging data yields more precise predictive outcomes.

Deep neural networks are often referred to as “black box” models due to the challenge in discerning the specific input components linked to predicted labels. Our goal is to illuminate the areas of focus within CNN that pertain to CT images and contribute to bone metastasis prediction. To achieve this, we employ the Grad-Cam method, which generates a visual representation of CNN’s attention through a heat map.

It is important to acknowledge certain limitations within our study. Firstly, our data emanates from a singular center, warranting the need for data from multiple centers to validate the model comprehensively. Secondly, the retrospective nature of the study introduces a degree of selection bias. Lastly, our study exclusively comprises Chinese participants, thus precluding the extrapolation of results to other ethnic groups.

In summation, deep learning emerges as a valuable technique for extracting pertinent information from CT images to predict bone metastases. Our approach of amalgamating clinical data and GBDT further refines prediction accuracy. Through validation and refinement in a larger and more diverse population, our multimodal model holds the potential to evolve into a pivotal auxiliary diagnostic tool in clinical practice.

Authors’ contributions

S.D.M. conceived the experiment, W.J.H. and W.J.Z. collected data and performed Data cleaning, H.B.J. and K.C. conducted the experiments and analyzed the results, S.D.M., W.J.H. and H.B.J. wrote the manuscript, W.J.H. and R.T.W. reviewed the manuscript.

Funding

This study has received funding by National Cancer Center Climb Plan (NCC201908B09) and Heilongjiang Provincial Postdoctoral Funding Project (LBH-Z15100).

Guarantor

The scientific guarantor of this publication is Shidi Miao.

Informed consent

Written informed consent was not required for this study because it was a retrospective study, informed consent from all participants was waived.

Ethical approval

Harbin Medical University Cancer Hospital Institutional Ethics Review Board approved this retrospective study.

Code availability

We provide the Python source code of DLR model training, which is freely available at https://github.com/ HotHeaven233/Deep-learning-predicts-BC-bone-metas tases.

Supplementary data

The supplementary files are available to download from http://dx.doi.org/10.3233/CBM-230219.