Deep learning approaches for breast cancer detection in histopathology images: A review

3.1Image pre-processing

In mammography, it is hard to identify the difference between normal glandular breast tissue and cancerous tissue. Furthermore, it is challenging to distinguish malignant tumors from the background in the thick breast tissue. There will be only a slight variation in the attenuation of the X-ray beam when it traverses normal glandular and cancerous breast tissue. Therefore, it is difficult to differentiate between them if it has not been preprocessed. Another issue with mammography is the quantum noise, which reduces the quality of the images. This is especially true for tiny entities that have poor contrast, such as a small tumor in a thick breast [68]. To circumvent this difficulty, contrast enhancement techniques are applied, which increase the visual quality of an image by improving the contrast between two objects, allowing one to readily detect the cancerous tissue. Other image pre-processing techniques commonly employed in histopathology images are image normalization and enhancement, image augmentation, image scaling, artefact removal, stain normalisation/removal, and so on.

Variations in imaging settings, such as differences in lighting, staining, and imaging instrumentation employed during image acquisition, can have an impact on the consistency of intensity values between images. Image normalisation is the process of converting the pixel values of images to a defined scale to modify the brightness and contrast [43]. This aids in the removal of any inherent variations produced by imaging conditions. Some of the frequently used normalisation techniques are Z-score normalisation, mean-standard deviation normalisation, and min-max normalisation. Also, various image enhancement techniques, such as contrast stretching, spatial filtering, histogram equalisation and noise filtering, could be used to improve the visibility of tissue structures in images [21]. These methods aid in enhancing the contrast of tissue features and suppressing noise, which makes it easier to spot structural changes that indicate the presence of malignancy.

The process of resizing an image to a specific size or resolution is known as image scaling. The histopathology images may have varying resolutions depending on the imaging environment [65]. Processing large, high-resolution images requires a significant amount of computing power. Therefore, it is essential to limit the size and resolution to a specific level without compromising the quality. Image scaling methods can be applied to fix this issue.

Training deep learning models requires a significant amount of training data, which may not always be readily available. To overcome this challenge, image augmentation [12, 57] can be used to generate new images from existing ones through various transformations such as rotation, horizontal and vertical flipping, cropping, adding Gaussian noise, translation, contrast adjustment, and more. By augmenting the data set in this manner, the amount of available training data can be increased, thereby improving the model’s ability to generalize and make accurate predictions on new, unseen data.

In the case of histopathology images, stain normalization is a crucial pre-processing step that helps compensate for the variability in staining that can arise due to differences in time and the person performing the process of staining [80]. Hematoxylin and eosin (H & E) tissue stains are the most frequently used tissue stains in histology. For instance, as seen in Figs 1 and 2, H stain clearly separates nuclei in blue against a pink backdrop of cytoplasm and other tissue areas. Although this makes it easier for a pathologist to identify and evaluate the tissues, these H & E stained images must be normalised [36] for automated image analysis due to varying lighting conditions while capturing a digital image and noise produced by the staining process. Stain normalization is a process used to reduce the impact of staining-related variations and ensure consistency in tissue characteristics across multiple images [53, 72, 8]. This technique aims to standardize the color and intensity of staining, thereby making the images comparable and facilitating reliable analysis.

3.2Segmentation

The segmentation process divides the image into numerous segments to isolate the region of interest from normal tissue and background [36]. Because of the poor contrast of the medical images, it is the most difficult task in automatic diagnosis systems [74]. The method for segmentation is chosen based on the different types of features to be extracted. To extract the target area in diseased images, several approaches such as region growing, nuclei segmentation, Otsu thresholding, etc. are utilised, along with filtering techniques like adaptive mean filtering, median filtering, and Wiener filtering. Thresholding is frequently done after background correction and filtering. Background correction uses an empty image to normalize the images [21].

In the case of histopathology images, a standard histology slide has a dimension of 15 mm × 15 mm tissue region that contains both essential and unwanted information [83]. Due to the high resolution at which these images are recorded, there will be an increase in both processing time and computational complexity. Segmentation is employed to locate and extract the areas that contain more specific information to process the images. Critical structures, such as malignant cells, tumor borders, and other key tissue components, are properly identified and isolated through segmentation, allowing for more precise analysis and feature extraction. Accurate segmentation allows CAD systems to detect minor discrepancies and changes in cell morphology, resulting in higher sensitivity for detecting early-stage cancer. Additionally, separating key structures decreases false positives and provides pathologists with improved diagnostic assistance. Segmented regions provide important information for tumor staging and prognosis by quantifying the size, shape, and spread of the tumor. Furthermore, it enables advanced analysis such as cell proliferation and morphological characterization, resulting in a better understanding of tumor biology. The fundamental process in the analysis of histopathology images is the segmentation of nuclei, which is the control center of the cell [84]. It contains DNA or genes, that give each cell instructions on how to behave, when to grow, when to die, etc. In cancerous cells, damage to the DNA of nuclei will affect the normal growth regulation of cells and result in malignancy. This emphasizes the significance of a thorough nuclei analysis. Moreover, precise quantitative characterization of the size, shape, and textural properties of nuclei is critical in histopathological image analysis. The most common types of segmentation are:

Imperfections in staining can cause fluctuations in tissue appearance in histopathological images, making nuclei segmentation in breast cancer imaging challenging [41]. Semantic segmentation combined with CNN makes complex mitotic images intelligible and can provide a lot of categorization information.

3.3Feature extraction and selection

Medical images contain a wealth of information, including subtle clues to pathology, irrelevant features, artifacts, and overlapping structures that can pose challenges for accurate interpretation. Such high-dimensional data can pose several challenges for automated algorithms. The technique of extracting features that are necessary for a given task from a set of features generated from raw data is known as feature extraction [27]. This approach will reduce computational complexity by eliminating noise and redundant information in the data. The new set of variables built through feature extraction should be capable of reconstructing the original data. One of the main issues is that using a large number of features on a small data set can lead to overfitting [37]. Overfitting occurs when the model is too complex and captures noise and random fluctuations in the data, rather than the underlying patterns and relationships. To address this problem, several techniques can be employed, such as feature selection, regularization, and dimensionality reduction [37].

Feature selection involves selecting a subset of the most relevant features based on their importance or relevance to the intended task. There are three commonly used methods for feature selection: filters, wrappers, and embeddings [17]. Filters are less computationally intensive but are slightly less accurate than the other two methods. Filters work by evaluating the relevance of each feature based on some statistical measure, such as correlation or mutual information, and selecting the top-ranked features. In contrast, wrappers and embeddings are more computationally demanding. The wrapper method selects features by evaluating the performance of a machine-learning model trained on different subsets of features. It can produce the best selection of features but requires training a model multiple times, which can be computationally expensive. Embedding techniques, such as Lasso and Ridge regression, select features by incorporating feature selection into the model training process. These techniques penalize the model for using irrelevant or redundant features, resulting in a more compact and accurate model. Overall, the choice of feature selection method depends on the specific requirements and constraints of the problem at hand, such as the size of the data set and the computational resources available.

Regularization methods, such as L1 or L2 regularization, penalize the model for using too many features, encouraging it to focus on the most important ones. Dimensionality reduction techniques, such as principal component analysis (PCA) or t-SNE, transform the high-dimensional data into a lower-dimensional space while preserving most of the relevant information [72]. This can be useful for visualizing the data, as well as reducing the computational complexity of machine learning algorithms that operate on high-dimensional data. However, it is important to note that PCA may not always be the best method for feature extraction, especially if the data has a nonlinear structure. In such cases, nonlinear dimensionality reduction techniques such as t-SNE [23] or Uniform Manifold Approximation and Projection (UMAP) [67] technique may be more appropriate.

Another challenge with high-dimensional data is the increased computational complexity. The learning algorithms may take a long time to train and make predictions when dealing with a large number of features [16]. To address this issue, several methods can be used, such as parallel computing, distributed computing, and model approximation. Parallel computing involves using multiple processors or cores to speed up the computation, while distributed computing involves distributing the computation across multiple machines. Model approximation methods, such as decision tree pruning or neural network compression, reduce the complexity of the model by simplifying its structure or reducing the number of parameters.

In breast cancer detection from histopathological images, the morphology of nuclei is a key factor to consider for disease diagnosis. To extract useful information from these images, various types of features need to be extracted using techniques such as morphological analysis, textural analysis, and graph-based analysis. Morphological features can be extracted to describe the size and shape of cells in the image, which can provide valuable information about the type and stage of cancer. Textural features, such as smoothness, coarseness, and regularity, can be extracted to reveal patterns and structures in the image that may be indicative of cancer. Graph-based topological features can also be extracted to describe the shape and spatial arrangement of nuclei in tumor tissue, providing insights into the characteristics of cancerous tissue [20, 50]. Once these features have been extracted, they can be utilized in the classification stage to distinguish between cancerous and non-cancerous tissue.

3.4Classification

The final stage of a computer-assisted detection system is classification, which involves categorizing a set of data into different categories or classes. The primary goal of classification is to determine the category into which a particular data point will fall. To achieve this, feature vectors extracted using feature selection techniques are used as input to the classification algorithm. Most classification frameworks consist of three phases: training, testing, and validation. During the training phase, the classifier uses available data to train the model [73]. The testing phase is used to predict the class of unlabeled data, and during the evaluation stage, the performance of the classification algorithm is assessed.

Breast cancer classification problems can be either binary or multiclass. Binary classification is used to differentiate between benign and malignant tumors, while multiclass classification can be used to classify the tumors into subtypes such as In-situ, Invasive, Normal, and Benign [42]. The input features for breast cancer classification can be derived from various sources, such as histopathology images or cytology data. These features can include morphological or textural features derived from nuclei after segmentation [8]. Various algorithms can be used for classifying data, including logistic regression, artificial neural networks (ANN), decision trees, K-Nearest Neighbors (KNN), Naive Bayes, Support Vector Machines (SVM), and Random Forests [30, 31]. The choice of classification algorithm will depend on the specific problem being addressed, the available data, and the performance requirements [32]. It is important to select the most appropriate algorithm and to fine-tune its parameters for optimal performance. By utilizing a well-designed classification algorithm, a computer-assisted detection system can accurately categorize new data and aid in making critical decisions in various applications, such as medical diagnosis and surveillance.

In recent years, deep learning methods have gained popularity and have been extensively used for classification tasks due to their ability to handle large amounts of data and their superior performance. DL systems employing transfer learning have emerged as a powerful technique for improving classification accuracy in breast cancer classification tasks [41]. Transfer learning involves pre-training a CNN on a large data set of images and then fine-tuning the network on a smaller data set of breast cancer images. This approach has been shown to improve classification accuracy compared to using a CNN trained from scratch on a small data set of breast cancer images.

Figure 4.

A general block schematic of various steps employed during computer-aided diagnosis in medical images.

5.1Artificial Neural Network (ANN)

Artificial Neural Networks (ANNs) are computing systems that are designed to imitate the biological neural networks found in the brain. At the core of a DNN lies the artificial neuron, which is a perceptron model composed of multiple interconnected layers. The three primary layers of a neural network are the input layer, hidden layer, and output layer. These layers work together to help classify input data and make predictions.

The effectiveness of an ANN is largely dependent on the number of hidden layers it contains. As the number of hidden layers increases, the performance of the ANN improves and the false positive rate decreases. However, this increase in performance comes at the cost of increased computational complexity. In an ANN, the input features are stored in the input layer, which is then projected into a higher-dimensional space by the hidden layer. The hidden layer processes the input features through a series of interconnected nodes, each of which computes a weighted sum of its inputs and passes the result through an activation function. This process helps extract more complex features from the input data, allowing the ANN to make more accurate predictions. Figure 5 presents the ANN illustration.

In a breast cancer diagnosis task, the goal is to classify samples into two categories: benign or malignant. To accomplish this, the input features are fed into the neural network’s input layer. The perceptron in the neural network processes the input attributes by passing them through the input layer, hidden layer, and output layer. Initially, each input is given a random weight, which indicates the significance of each input variable. Additionally, each perceptron has a bias value, which is a numerical value. An activation function processes each perceptron, determining whether or not the perceptron should be activated. Only activated perceptrons transmit data from the input to the output layer.

The output layer calculates the probability of the data being either benign or malignant. If the expected output is incorrect, the neural network is trained using the back propagation method. During back propagation, the actual results are compared to the predicted results, and the weights of each input are adjusted to minimize the error. This process leads to more precise results, improving the accuracy of the neural network’s predictions.

Figure 6.

Illustration of basic blocks in a convolutional neural network.

5.2Convolutional Neural Network (CNN)

Recently, CNNs have been widely used in breast cancer diagnosis as they can identify patterns and features in images, allowing for accurate classification. As shown in Fig. 6, a CNN typically consists of four main layers: the convolutional layer, ReLU layer, pooling layer, and fully connected layer. The convolutional layer is designed to detect spatial patterns or features in the input data. It uses a set of learnable filters or kernels to convolve over the input, performing element-wise multiplications and aggregating the results. This process helps capture hierarchical features, preserving spatial relationships. The ReLU layer introduces non-linearity to the network. After the convolutional or fully connected operations, the ReLU activation function is applied element-wise to the output. It replaces all negative values with zero, allowing the model to learn complex patterns and relationships in the data. ReLU aids in the network’s ability to capture non-linearity. Pooling layers are used to downsample the spatial dimensions of the input volume. Common pooling operations include max pooling and average pooling. Pooling helps reduce the spatial resolution, retaining important features while discarding less significant details. The fully connected layer, also known as the dense layer, connects each neuron to every neuron in the previous and subsequent layers. It transforms the features learned by the previous layers into a format suitable for classification or regression. The output of the fully connected layer is often fed into a softmax activation function for classification tasks or a linear activation for regression tasks.

The use of CNNs in breast cancer diagnosis allows for more accurate and efficient analysis of medical images. However, CNNs require a large amount of labeled data to achieve good performance, which can be challenging to obtain in some cases. One way to address this issue is to use a technique known as transfer learning. Transfer learning is a method that utilizes a pre-trained model to solve a different but related problem. The pre-trained model has already learned a set of features from a large data set, making it easier to train on a smaller data set with a similar problem. Transfer learning is especially useful when the amount of available data for training is limited. In a CNN, the deeper layers learn task-specific attributes, while the shallower layers learn more basic features such as edges, patterns, etc. However, these shallow layers are harder to train due to vanishing gradients. Transfer learning takes advantage of this by freezing the earliest layers and changing only the final few layers according to the specific task. This allows for the transfer of knowledge from the pre-trained model to the new task. Pre-trained models like VGG-16, ResNets, and DenseNets have been trained on massive data sets and can be used as a starting point for transfer learning. By modifying the final layers of these models, they can be applied to more specialized tasks, such as fine-grained classification or object detection. Figure 7 shows the schematic of the transfer learning process.

Figure 7.

Schematic of transfer learning process.

Figure 8.

The basic structure of an autoencoder network.

5.3Autoencoders

Autoencoders are a type of neural network architecture that can learn to compress data and then reconstruct the compressed data back to its original shape and size. They consist of three layers: an input layer, a hidden layer, and an output layer. The hidden layer, also known as the bottleneck layer, is where the data is compressed.

An autoencoder works in two stages: encoding and decoding. During the encoding stage, the input data is compressed into a smaller dimension in the hidden layer. This is achieved through a series of mathematical operations that transform the input data into a lower-dimensional representation. This compressed representation is then stored in the hidden layer. During the decoding stage, the compressed representation is used to reconstruct the original input data. The hidden layer’s output is transformed back into the original input space through another series of mathematical operations that reverse the encoding process. The final output is compared to the original input, and the autoencoder’s performance is evaluated based on how accurately it can reconstruct the input data. Autoencoders are trained using the backpropagation technique, where the difference between the input data and the reconstructed output is used to adjust the network’s weights. This process is repeated until the autoencoder can accurately reconstruct the input data. Figure 8 shows the basic structure of an autoencoder network.

Autoencoders consist of an array of nodes in the input, hidden, and output layers. In order to feed an input image to the input array of nodes, the image must first be transformed into a one-dimensional array. This array is then encoded into a hidden representation in the bottleneck layer. An important goal of an autoencoder is to ensure that it can accurately reconstruct the input while avoiding overfitting or memorizing the training data. To achieve this, a loss function is used that considers both the reconstruction error and a regularizer term. The reconstruction error measures the difference between the input image and its reconstructed output, while the regularizer term tries to make the autoencoder insensitive to input. The regularizer term in the loss function encourages the autoencoder to learn from the hidden representation rather than directly from the input. By doing so, the autoencoder only learns the essential features necessary for reconstructing the input image, rather than simply memorizing the training data. This helps to prevent overfitting and improve the autoencoder’s ability to generalize to new, unseen data.

Figure 9.

The concept of generative adversarial network (GAN).

5.4Generative Adversarial Networks (GAN)

GAN is a type of deep learning model consisting of two sub-models, namely the Generator model and the Discriminator model. The main objective of the generator model is to create synthetic data that mimics the real data, while the discriminator model aims to distinguish between the real and fake data produced by the generator. The discriminator model is typically a convolutional neural network (CNN) with multiple hidden layers and a single output layer that produces a binary output of either 0 or 1. A value of 1 indicates that the provided data is real, while a value of 0 indicates that the data is fake. On the other hand, the generator model is an inverse CNN that takes a random noise input and transforms it into a sample from the model distribution. In other words, it generates synthetic data from a piece of input data.

During the initial stages of training, the generator produces data that is very different from the real data, making it easy for the discriminator to detect it as fake. However, as training progresses, the generator starts producing fake data that is increasingly similar to the real data, making it more difficult for the discriminator to distinguish between the two. Eventually, if the generator training is successful, it will produce data that is a perfect match for the real data, and the discriminator will begin to categorize the fake data as real. This means that the discriminator’s accuracy will decline, indicating that the generator has successfully learned to generate synthetic data that is indistinguishable from the real data.

In medical imaging applications, collecting enough labeled data for training deep neural networks can be challenging. GANs can be used to generate synthetic data with a probability distribution that mimics that of benign samples, providing a useful tool for augmenting training data and improving the accuracy of medical image classification tasks. The concept of GAN is illustrated in Fig. 9.

7.1Confusion matrix

A confusion matrix is a tool utilized to assess the effectiveness of a classification model by analyzing the predicted and actual class labels of a group of test samples. It presents a concise representation of the number of true positive (P1), true negative (N2), false positive (P2), and false negative (N1) predictions made by the model. A sample confusion matrix is shown in Fig. 10. It is a table that classifies predictions based on how closely they correspond to the exact value. It can be used to determine the ROC curve, recall, specificity, accuracy, and other metrics.

Figure 10.

Schematic of a confusion matrix showing true positive (P1), true negative (N2), false positive (P2), and false negative (N1) cases.

7.2Accuracy

It is a measurement of how many classes across all classes are accurately predicted. Accuracy should be valued as highly as possible.

7.3Precision

Precision is a classification performance metric that measures a model’s ability to correctly identify positive cases. It is the proportion of true positives (P1) to the total number of predicted positive cases (P1 + P2), expressed as a percentage or a decimal between 0 and 1. Precision is calculated as:

7.4Sensitivity

Sensitivity is a classification performance metric that measures a model’s ability to correctly identify positive cases. It is also known as the true positive rate (TPR) and is the proportion of true positives (P1) to the total number of actual positive cases (P1 + N1), expressed as a percentage or a decimal between 0 and 1. Sensitivity is expressed as:

7.5Specificity

Specificity is a classification performance metric that measures a model’s ability to correctly identify negative cases. It is the proportion of true negatives (N2) to the total number of actual negatives (N2 + P2) and is expressed as a percentage or a decimal between 0 and 1. Specificity is expressed as:

7.6F1 score

The F1 score is a classification performance metric that merges precision and recall into a single score. It is calculated as the harmonic mean of precision and recall and ranges between 0 and 1, with higher values indicating better model performance.

Precision quantifies the proportion of true positives to the total number of predicted positives, while recall measures the proportion of true positives to the total number of actual positives. The F1 score equally emphasizes both precision and recall, making it useful in evaluating models where both measures are critical.

The F1 score is particularly valuable when dealing with imbalanced data sets, where one class has a much larger number of observations than the other. In such cases, the F1 score is a more reliable measure and is represented as:

7.7ROC curve and AUC

The ROC curve is a graphical plot that visualizes the effectiveness of a binary classification model across different classification thresholds. It compares the true positive rate (TPR) to the false positive rate (FPR) for a range of thresholds. A model with a higher TPR and lower FPR is considered better. The ROC curve also highlights the balance between TPR and FPR, with the area under the curve (AUC) being a metric of overall performance. A perfect model has an AUC of 1.0, whereas a random model has an AUC of 0.5.

AUC is widely used to compare different classification models because it is robust against imbalanced data, unlike accuracy, precision, and recall. Higher AUC implies better model performance in distinguishing between positive and negative classes. In a ROC curve, the X-axis represents the FPR, and the Y-axis represents the TPR, as illustrated in Fig. 11.

By lowering the classification threshold, more objects are classified as positive, increasing both true positives and false positives. As the ROC curves move towards the top-left corner of the ROC space, the accuracy of the classifier improves. This is because the classifier has a higher TPR and a lower FPR, indicating that it is better at correctly identifying positive cases while minimizing false positives.

Figure 11.

A sample ROC curve that visualizes the effectiveness of a binary classification model across different classification thresholds.

7.8Patient score

The patient score is a metric used to evaluate the performance of a classification model in detecting diseased images for individual patients. It is calculated by multiplying the total number of diseased images for a patient (Np) by the proportion of diseased images that the model correctly recognized for that patient (N𝑟𝑒𝑐).

The patient score reflects how well the model performs at identifying abnormal images for each patient. A higher patient score indicates that the model correctly recognized a larger proportion of diseased images for that patient, while a lower score suggests that the model may have missed some diseased images. It is expressed as:

7.9Patient recognition rate

It is the ratio of the sum of patient scores to the overall patient count.