Smart e-waste management system utilizing Internet of Things and Deep Learning approaches

3.1.System architecture

Figure 1 illustrates the overall architecture of our proposed system. It uses 1) a Raspberry Pi 4 microcontroller for performing e-waste object detection and identification, 2) an Electron Particle kit with 3-G cellular connectivity that is responsible for: a) continuously monitoring the condition of the e-waste bin, such as the temperature, and humidity inside the bin, to detect and prevent fires from happening, b) providing up-to-date real-time information to different stakeholders (i.e., municipalities, service waste manager, system administrator) regarding e-waste count, e-waste weight/volume, and the filling level of the bin.

Fig. 1.

Proposed smart e-waste management system architecture.

This project uses the TensorFlow Lite application programming interface (API) to run the model trained on an electronic object dataset for image e-waste object detection. In addition, we utilized the ThingSpeak cloud platform to store and retrieve all the monitoring data using two main protocols: Hypertext Transfer Protocol (HTTP) and Message Queuing Telemetry Transport (MQTT) protocol over the Internet. An interactive Web-based interface displays stored data to system administrators.

3.2.System components

Table 2 lists all the components and modules used in our project, with their corresponding functionalities. These components work together to form an integrated system to achieve smart, green, healthy, and sustainable environments and cities. Regarding the interactive mobile applications for collector trucks and users, we will defer their development for future implementation.

3.3.System workflow

This subsection describes the working mechanisms of the proposed smart e-waste bin as follows:

The ultrasonic, infrared, and load sensors will only work when e-waste is disposed of. In addition, we will send the values of humidity and temperature sensors to the ThingSpeak platform only when the difference between the current and the last reading increases above a threshold value. The utilized approach mimics an energy-efficient way to operate the smart e-waste bin.

3.4.Prototype implementation (hardware)

The smart system was successfully developed as a proof-of-concept in this work, and it could be a promising step toward enhancing e-waste management. Figure 7 shows a schematic diagram for a part of the proposed smart e-waste bin. The diagram presents some hardware used to operate the smart e-waste system, a Particle Electron Kit with four main sensors, namely HC-SR04 ultrasonic sensor, DH11 humidity and temperature sensor, IR Flame module, and Infrared obstacle sensor. The main aim of this part is to measure the filling level of the trash bin and keep track of the amount of e-waste being deposited. In addition, predicting potential fire incidents or unfavorable events, such as a significant increase in the temperature within the bins.

Fig. 7.

Schematic diagram for a part of the proposed smart e-waste bin, which shows a Particle Electron Kit with four main sensors.

3.5.System cost

Table 3 outlines the total cost of developing the proposed system. This cost is reasonable given that the system reduces the need for manual labor in traditional waste management practices, resulting in decreased overall waste management costs.

4.1.Cloud, fog, or local computing

Recently, various IoT projects utilized cloud computing and fog computing due to the limitation of computing power and the battery life faced by current technologies. Both approaches can shift the vast amount of data and computation previously processed locally in the device to the fog or remote cloud nodes. Compared to cloud computing, fog computing achieves superior latency and faster response time in IoT systems [37].

However, performing fog computing by transmitting images for e-waste object detection to the fog node is not suitable for our project for the following reasons: For our proposed approach, we will utilize GSM communication for data transmissions, which has the desirable range characteristics (i.e., several kilometers between the GSM module and the tower) suitable to be implemented in the e-waste trash bin since these bins will be scattered along with the city. However, since the Pi camera in our project will capture a 5MP image of e-waste objects with a file size of 15 MB, fog computing will not be suitable in our case since GSM has limited data rates per channel. Consequently, this will result in higher latency and reduce the response time compared with the processing on the device. Table 4 illustrates the typical transfer time to upload an image of 15 MB in size via different GSM standards, further emphasizing the inefficiency of offloading processing tasks to external fog nodes.

In contrast, on-device processing offers numerous efficiency and cost-effectiveness advantages. It reduces latency and increases response time, lowers data usage with limited bandwidths, enhances privacy and security by keeping data locally, and ensures better offline functionality in areas with limited connectivity. Moreover, on-device processing can lead to significant cost savings, as it eliminates the need for costly server infrastructure and cloud services, making it a compelling approach for our objectives [5].

In addition, with the development of TensorFlow Lite (TFLite), we can convert and optimize any object detection model for deployment on mobile devices such as raspberry pi and Android. Thus, processing object detection on the local device is more resource-efficient and assists in making data-driven decisions faster.

We will use the cloud platform mainly to store the collected data (i.e., filling level, e-waste object count, e-waste object type, bin weight, humidity, temperature, etc.) and perform further analysis (e.g., predictive models).

4.2.Choosing a deep-learning framework

In our project, we have chosen TensorFlow Lite for one main reason. A typical object detection model typically requires a powerful Graphical Processing Unit (GPU) to perform efficiently. However, such a requirement is unsuitable for a smart e-waste management system that utilizes a low-power embedded mobile device like a Raspberry Pi. Fortunately, TensorFlow Lite enables the use of object detection models on these constrained resources.

TensorFlow Lite is a deep learning framework that is open-source and can be used across different platforms. It transforms pre-trained models into an optimized portable format called FlatBuffers, identifiable by the .tflite file extension, that can be tailored for speed or storage [24]. In essence, TensorFlow Lite is a collection of tools that facilitates on-device machine learning, particularly on devices with limited computational and memory resources. Consequently, developers can execute their models on mobile, embedded, and IoT devices, including Linux-based embedded devices like Raspberry Pi or microcontrollers. Since models at resource-constrained devices need to be lightweight and have low latency, the amount of computation required for prediction must be reduced. TensorFlow Lite addresses this challenge by employing two optimization methods: quantization and weight pruning. Quantization involves reducing the precision of the numbers used to represent various TensorFlow model parameters, thereby reducing the model’s weight (e.g., 32-bit floating points weights can be converted to 8-bit floating or integer points). On the other hand, weight pruning involves removing parameters within a model that have minimal impact on the model’s performance [21].

TensorFlow Lite model addresses five key constraints: 1) low latency by eliminating the round-trip from the device to the server, 2) privacy protection by ensuring that personal data is not shared across the network, 3) no need for permanent internet connectivity, 4) reduced model and smaller binary size, 5) efficient power consumption as the inference is made locally irrespective of network connectivity [15].

4.3.Choosing an architecture for object detection

To identify the most effective model architecture for detecting e-waste, three latest cutting-edge, single-stage object detection deep learning models are deployed and compared, YOLOv5s, YOLOv7-tiny and YOLOv8s.

The authors of YOLO [28] have introduced a novel approach to object detection by reframing the problem as a regression task rather than a classification task. They utilize a convolutional neural network to simultaneously predict both the bounding boxes and class probabilities for all objects present in an image, hence they have named it as You Only Look Once (YOLO). Compared to other single-shot detectors, YOLO generally achieves superior performance in terms of both speed and accuracy [7]. YOLO has undergone several revisions (YOLOv2 through YOLOv8) that have resulted in improvements in various areas, including modifications to the loss function, changes to network design, scaling of input resolution, and adaptations to anchor boxes [34].

YOLOv5 represents an evolutionary step within the YOLO family of object detection models. This updated version incorporates a Cross Stage Partial Network (CSPNet) backbone and integrates a path-aggregation neck (PANet). Unlike previous YOLO iterations that were modifications of the original Darknet framework, YOLOv5 is built entirely from scratch using PyTorch. Furthermore, it incorporates mosaic data augmentation and automatic learning of bounding box (BB) anchors to enhance the speed and accuracy compared to previous versions [23].

YOLOv7 is a recent YOLO version that was released in July 2022. According to the official YOLOv7 article [36], it is the fastest and most accurate real-time object detection model for computer vision tasks. YOLOv7 surpasses all known object detectors in both speed and accuracy in the range from 5 FPS to 160 FPS and has the highest accuracy 56.8% AP among all known real-time object detectors with 30 FPS or higher on GPU V100.

YOLOv8 is the newest state-of-the-art YOLO model that can be used for object detection, image classification, and instance segmentation tasks [33]. It was developed by Ultralytics, who also created the influential and industry-defining YOLOv5 model. YOLOv8 features a new backbone network and a design that makes it easy to compare model performance with older models in the YOLO family. It is designed to be fast, accurate, and easy to use, making it an excellent choice for a wide range of object detection and tracking, instance segmentation, image classification and pose estimation tasks [18].

4.4.Dataset collection and preparation

To train the pre-trained object detection model, we have collected around 4000 images along with their .txt label files of four electronic object types, namely Monitor, Keyboard, Mouse, and Headphone, from the Open Images Dataset V7 [25]. Open Images Dataset is a collaborative release that comprises around nine million images annotated with image-level labels, object bounding boxes, object segmentation masks, localized narratives and visual relationships. Regarding the .txt label files, they contain the coordinates of objects bounding boxes with their associated classes.

To ensure equal e-waste class coverage, we collected one thousand images per e-waste type with different backgrounds, orientations, and lighting conditions. In this study, we randomly selected 75% of the original images as the training set (3000 images), 15% as the validation set (600 images), and 10% as the test set (400 images). Additionally, we used stratified sampling to preserve the percentage of samples for each class in the training, testing and validation data.

In order to prepare the images for model training, some pre-processing steps are needed. First, auto-orientation is applied to straighten out any tilted or off-axis images. Second, the images are resized to meet the 640×640-pixel requirement of the YOLOv5s, YOLOv7-tiny and YOLOv8s models. Figure 8 presents some examples of annotated images from the dataset.

Fig. 8.

Some annotated images from the Open Images Dataset v7.

4.5.Training phase

Experimental setup In this study, we carried out model training on Google Colab. Google Colab is a cloud-based service that utilizes Jupyter Notebook for machine learning tasks. It offers a runtime environment specifically tailored for deep learning and provides free access to powerful GPU resources. We used the concept of transfer learning to simplify training YOLOv5s, YOLOv7-tiny and YOLOv8s models using a custom dataset.

Transfer learning is a technique that takes an already trained model for a related task as input and uses it as the starting point to create a new custom model [14]. Retraining a model with a custom dataset reduces the required training data and will shorten the training time. In addition, transfer learning enables object detection even if the dataset is small.

Before training, the image size is set to 640, and a batch size of 16 is configured. The model is initialized with pre-trained weights from YOLOv5s, YOLOv7-tiny and YOLOv8s. The training process is conducted for a total of 100 epochs. After training, the YOLO model is optimized for Raspberry Pi usage through a conversion process. Initially, the model is converted to TensorFlow format using ONNX (Open Neural Network Exchange), and subsequently, it is further converted to TensorFlow Lite (TFLite). This ensures compatibility and efficient deployment on Raspberry Pi for use in object detection. Figure 9 shows the pipeline for developing the TensorFlow Lite object detection model needed for the proposed system.

Fig. 9.

Pipeline for developing the e-waste TensorFlow Lite object detection model.

Evaluation metrics The evaluation of object detection models involves various metrics to assess their performance. We utilized the following metrics in our evaluation:

These metrics collectively assess the accuracy, detection capability, and overall performance of object detection models, enabling a comprehensive evaluation of their effectiveness in identifying and classifying objects.

Furthermore, loss functions such as box loss and classification loss are employed to assess the model’s utilization and retention of data.

4.6.Training / object detection results

Loss functions In the analysis of YOLOv5s, YOLOv7-tiny and YOLOv8s models, the behavior of box loss and classification loss for the training set as the number of epochs increases is examined. All algorithms show a decreasing pattern, indicating that they effectively model the training dataset.

Fig. 10.

The loss functions over 100 epochs of YOLOV7 and YOLOV8 models.

The box and classification losses for the validation set also demonstrate a similar decreasing trend in YOLOv5s and YOLOv7-tiny, suggesting that the model improves its ability to predict bounding boxes and classify objects accurately. This positive trend signifies the model’s learning process. By extending the number of epochs, there is potential for further refinement and fine-tuning of the model. However, in the case of YOLOv8s, it is observed that the loss functions reach a point where they either stabilize or gradually increase during training, indicating the presence of an overfitting problem. With epoch = 100, the loss functions are summarized in Fig. 10. Fortunately, in YOLOv5s, YOLOv7-tiny and YOLOv8, we can access the best-performing version (model weighs with the highest performance) for later use in evaluation or deployment.

Performance metrics With each epoch, as the loss parameters of the model gradually decrease towards their minimum values, there is generally a corresponding improvement in precision, recall, and evaluation metrics such as mAP@50 and mAP@50-95 scores. This relationship is depicted in Fig. 11, where the upward trend of these metrics demonstrates the model’s increasing performance and ability to accurately detect objects.

Fig. 11.

The performance metrics over 100 epochs of YOLOv5s, YOLOv7-tiny and YOLOv8s models.

Table 5 summarizes the overall performance metrics of YOLOv5s, YOLOv7-tiny and YOLOv8s, as well as the performance of specific classes for each model on the validation set.

According to Table 5, YOLOv5s got the highest precision and lowest recall. Having the highest precision means that the model has a low rate of false positives; It is cautious in making predictions and tends to be more accurate when identifying an object. On the other hand, having the lowest recall means that the model may miss some objects or have a high rate of false negatives; It is more likely to overlook particular objects or fail to include them in its predictions. On the other hand, YOLOv8s achieved a good balance between both metrics with the highest recall; This signifies that the model can accurately identify many objects while minimizing the number of missed detections.

Comparing the mAP@50 values, YOLOv8s and YOLOv5s achieved a comparable score of 0.72 and 0.723, respectively, whereas YOLOv7-tiny attained a value of 0.705. Similarly, when considering the mAP@50:95 values, YOLOv5s and YOLOv7-tiny achieved 0.49 and 0.481, respectively, while YOLOv8s achieved a higher score of 0.518. These results highlight that YOLOv8s outperforms others in terms of both metrics, which indicates that YOLOv8s demonstrates an improved ability to detect objects accurately across a broader range of IoU (Intersection over Union) thresholds. Similar performance results are achieved on the test set (refer to Table 6).

Confusion matrix In order to gain a detailed understanding of the model’s predictions for different classes, we can visualize the confusion matrix. Figures 12 to 14 display the confusion matrices for YOLOv5s, YOLOv7-tiny, and YOLOv8s, which provide a distribution of all predictions during the validation stage of each model.

Fig. 12.

Confusion matrix of YOLOv5s model.

Fig. 13.

Confusion matrix of YOLOv7-tiny model.

Fig. 14.

Confusion matrix of YOLOv8s model.

By examining the confusion matrices, we can observe that the majority of predictions fall along the main diagonal, which indicates that the models have successfully classified the ground truth objects and their corresponding predicted counterparts for e-waste classes. Interestingly, as we progress to the subsequent versions of YOLO, namely YOLOv5s, YOLOv7-tiny, and YOLOv8s, we observe improvements along the main diagonal of the confusion matrices. This improvement indicates enhanced accuracy and effectiveness in correctly detecting e-waste objects across these successive versions of YOLO.

However, there are a few instances where predictions deviate from the matrix’s main diagonal, suggesting misclassifications. These outliers indicate cases where the model may have incorrectly assigned a different class label to particular objects or misjudged the localization of the bounding boxes.

In particular, our object detection models sometimes incorrectly predict the presence of an object (e.g., a monitor) where there is only background or vice versa, which can arise from variations in object appearances and complexities in the background. Moreover, there are minor misclassifications between the computer mouse and headphone classes, which can be attributed to some similarities in appearance or contextual factors. Therefore, we may need to collect more diverse training images or apply various data augmentation techniques to diversify the training dataset, which will be left for future work.

Figure 15 shows some predictions made by YOLOv5s, YOLOv7-tiny and YOLOv8s on the test dataset. According to the results, YOLOv8s demonstrated a higher prediction accuracy than others. This observation suggests that YOLOv8s has successfully learned the visual features and patterns specific to electronic objects, allowing it to make more precise predictions.

Fig. 15.

The detection results of YOLOv5, YOLOv7 and YOLOv8s on some samples of the test set.

Table 7 summarizes the training time and the obtained size for each model. It is observed that YOLOv7-tiny has the smallest model size, whereas YOLOv8s has the largest model size. Conversely, YOLOv5s demonstrates the shortest training time, while YOLOv7-tiny exhibits the longest training time.

Based on the previous observations, the study selected YOLOv8s as the baseline object detection model and used it for subsequent experiments. The YOLOv8s model underwent a conversion process to make it compatible with a Raspberry Pi device. The process involved translating the PyTorch model weights into the ONNX format using PyTorch’s ONNX export function. Once converted to the ONNX format, the model was further transformed into the TensorFlow Lite (TFLite) format. TFLite is specifically designed and optimized for mobile and embedded devices, ensuring improved speed and efficiency. This conversion process allowed the YOLOv8s model to be seamlessly integrated into the Raspberry Pi, enabling efficient object detection on the device. According to Table 8, we got a minor reduction in the performance metrics.