Using IoT and AI to replenish household food supplies: A systematic review

1.Introduction

The waste of food is an essential issue to consider. There are many reasons for the waste. Without an organized shopping list, consumers purchase food based on imperfect memories of what remains at home and advanced marketing techniques used by supermarkets and other vendors. Additionally, severe weather conditions (or a pandemic in some cases) that require home isolation may make people unintentionally buy more than needed. People may forget their uneaten food, be unaware of expiration dates, store leftover food for too long, and food not well organized inside the refrigerator—losing food has many consequences. As part of home appliances, this paper intended to focus on refrigerators in particular as they always have perishable items that require monitoring. In contrast, other home appliances, such as washers and dryers, are occasionally used.

Inside the refrigerator, wasted food will unnecessarily consume space and electricity; the spoiled food means the water, labor, and other resources needed to produce it have been wasted [6,20,34,41,52,61]. A specialized agency of the United Nations called the Food and Agriculture Organization (FAO) studied food that ends up being wasted worldwide. The FAO study shows that one-third of the total amount of food worldwide is thrown away or lost, and consumers cause 40% of the waste [20,94]. The study was based on related kitchen appliances and the storage used to store food [20]. Adding features that track stored food items, monitor consumption and remaining food levels, will reveal a pattern of household behavior and consumption. The pattern will form the household replenishment system that will provide a shopping list to guide the household and stores. The historical information regarding the consumption rate of each perishable item will be gathered and recorded by the implanted sensors in the storage compartment. Then, it will be analyzed accordingly to form the pattern and predict the upcoming purchase along with the expected quantity [53,87].

Artificial Intelligence technology’s original seeds were planted in 1950 by Alan Turing [85]. Mr. Turing, proposed his idea in his book “Computing Machinery and Intelligence,” wherein he suggested people be concerned about the question “Can machines think?” [85]. From the proposed question by Mr. Turing, the basic idea of AI can be understood. It allows computers and machines to process data to be useful information using AI tools and techniques such as text mining, data mining, machine learning, image recognition, image processing, voice recognition, natural language processing, and other tools that allow machines to think, make and help decision-making processes like humans [34]. For example, Shweta in 2017 presented a solution for the smart refrigerator by using Artificial Intelligence technology [81]. The presented solution uses a machine-learning approach called the Aging algorithm. It includes a database to record the data processed by a microcontroller using the algorithm. The database is pre-loaded with images of different types of vegetables to train the data. The data was trained to detect the texture, shape, and other features of the captured images. It allows the system to compare the captured images with the loaded images. The comparison takes place to make a proper decision on the vegetable’s age. The system aims to monitor only vegetables. Thus, a camera is placed inside the refrigerator where the vegetables are located. The camera captures images and is connected to the microcontroller, where the images get analyzed for their age. After the analysis, the images will be sent to a microprocessor that converts and transmits the received information into signals. The signals will be received in the form of voice. Then, an attached voice indicator notifies the household about the vegetable status [81].

Nagarajua et al. [59] proposed a conceptual idea of using artificial intelligence in smart refrigerators to eliminate the wastage of food. This paper does not include specific components to be used; it was kept for the implementer to decide. The proposed solution has two approaches. One approach is using image recognition, referred to as predictive vision. Image recognition is an approach that could be applied by one of the AI tools called deep-neural networking. The authors suggested using the conventional neural network for vast data in this paper. The captured images will be processed, and the system will make the decision based on a comparison with pre-loaded identified images in a database and trained data. The second approach is using Natural Language Processing (NLP), which allows the household to communicate with the smart refrigerator by speaking out loud whenever they place items or take out items into or from the refrigerator; quantities of items can also be mentioned in the refrigerator. NLP will recognize the spoken information by the household and record it in a database. The database will be updated every time the household gives any voice command [59].

Gull et al. [27] proposed a conceptual idea for a system to enable the household to monitor food items stored in a smart refrigerator using Artificial Intelligence of Things (AIoT) via a personal computer. A novel idea of embedding the AI. Using decision tree advanced ID3 model, a machine learning algorithm, and an eNose system with different gas sensors. The gas sensors are used to identify food items, meat, rice, fruits, and vegetables from the emitted gases. Then, the captured data is sent to a trained database to label the food items accordingly [27]. Therefore, the camera module, eNose system, voice recognition, and Natural Language Processing technologies could mimic human action. The obtained data could be recorded and processed to train prediction models. Upon verification and validation of the developed models. The models would help to make proper decisions in predicting the suggested shopping lists and provide the desired output.

The state-of-the-art Internet of Things (IoT) technology made it easier to connect things (objects and electronic devices) to the Internet. IoT technology allows objects to communicate, exchange, and store information via communication channels, databases, and internet protocols that make up an IoT platform. The IoT is driven by AI technology. The IoT started in 1982 at the University of Carnegie Mellon. Computer science students modified a Coke machine so it would inform a computer network about the inventory status and drink temperature. The IoT revolution began in earnest in the early 1990s [21,40,84]. Just as the Carneige-Mellon students found Coke machine data useful, recent technological developments, as shown in this literature review, will bring accurate, timely information to consumers to automate their household replenishment systems. Information can be obtained by involving the new technology in home appliances. A patent titled “Household consumable item automatic replenishment system including intelligent refrigerator” granted to Sone in 2001 demonstrated an automated household replenishment system. It included various components that are interconnected and connected to the Internet to monitor stored items’ status. A detailed diagrams and illustrations were provided to illustrate the workflow of the proposed system [83]. Thus, it is essential for an easier lifestyle to maintain household demands along with proper management of supplies in a smart structure. The potential of IoT would lead to smart homes, markets, industries, and eventually smart cities [3].

Realizing that information flow can arise after things have been identified and devices are connected in the same network, called Information integration, leads to thinking about making that possible. The literature found that exchanging data can be done in a system and stored in a shared database such as Google Firebase or an offline database. The data can be retrieved as input from sensors, RFID systems, camera modules, and barcode systems in many ways. A controller such as the Arduino UNO dashboard device can be configured to retrieve data from different sources, process them, and send them to a mobile application or a central database as an output [9], as shown in Fig. 1. The database is accessible by authorized people via the Internet. The information stored in this database can be viewed, managed, modified, and updated for several reasons. From the user’s perspective, the information could be about the flow of the stored items at home, allowing the households to make proper decisions about what to purchase next, know what items are about to expire, and control the surrounding climate to ensure freshness. Furthermore, if stores and suppliers are allowed to interact with that database, they can analyze the habits of each household, improve products, set proper marketing plans, form a pattern of subsequent purchases, and ensure the product availability [88].

Fig. 1.

Data flow of household replenishment system.

Most current devices can be identified and connected to the internet wirelessly via WiFi or Bluetooth connections or wired via Ethernet for exchanging information and other purposes. Appliances such as regular refrigerators or storage cabinets require specific AI or IoT components to allow them to be converted into smart storage compartments of perishable items.

This paper will focus on AI and IoT components and information integration methods used by the reviewed papers that enhance the household replenishment system. Therefore, this paper analyzes relevant research about how AI and IoT technologies can create a unified framework that includes all used components to eliminate food waste and enhance the Household Replenishment System and provide future directions. The presented possible combination of all used components could be used as guidance for developers and future directions for researchers.

The remainder of this systematic review is organized as follows; section two is the research methodology, section three is the Household Replenishment Systems Analysis, section four is the findings and discussions, and the fifth section is the conclusion and future directions.

2.Research methodology

The comprehensive research has followed the PRISMA guidelines, as shown in Fig. 2, which are helpful for researchers creating and arranging systematic reviews [61].

Fig. 2.

Search strategy following PRISMA flow diagram 2020.

Several factors cause the slow movement of research and implementation of the new technologies needed to convert storage compartments into smart, home storage compartments for perishable items. Among them are the cost of system configuration, manufacturing, and research and security issues [61].

This systematic review surveys the relevant work on an application of IoT, Artificial Intelligence of Things (AIoT) technologies on perishable items storage compartments, the household interaction, and a detailed description of components, 2) point out the advantages and limitations of the concept, 3) find the frequency of the components appear in the research, 4) and perform data mining technique that built a decision tree for a continuous variable decision on the reviewed papers, and 5) offers a unified framework to developing a system that can include all necessary components. Also, future research directions to enhance the household replenishment system to reduce food wastage will be provided, as shown in Fig. 3.

Fig. 3.

Summary of the systematic review methodology.

This systematic review focuses on papers published from 2000 when LG Electronics invented R-S73CT the first smart refrigerator, and research on technology involvement in refrigerators began to increase [23,91]. Combination of keywords were used for the search, as follows: “Smart refrigerator”, “Smart refrigerator AND “internet of things”, “Smart refrigerator” AND “artificial intelligence”, “Smart Fridge”, “Grocery Ordering Systems”, “Fridge” AND “IoT” AND “AI” AND “Smart refrigerator”, “smart food storage”, “Intelligent Refrigerator”, “Intelligent Refrigerator” AND “artificial intelligence”, “Intelligent Refrigerator” AND “internet of things”. The search conducted in this paper uses ProQuest and Google Scholar databases. Only papers written in English and published in journals and conferences are selected for this paper. Publications were excluded if they had or were: 1) access restrictions for the entire paper, 2) papers for class projects, 3) proposed systems that applied IoT or AI on other than home food storage compartments, 4) relevant but the mechanism of household interaction was missing, and 5) newspapers, wire feeds, blogs, podcasts, websites, patents, dissertations and theses, books, and magazines are excluded. Table 1 shows a summary of the search results using the keywords and search engines.

The initial search resulted in 5800 possible papers to review, as shown in Fig. 2. Before the initial screening 226 duplicated results were removed. From the initial screening of the titles and abstracts of each result and applying the exclusion criteria, 72 papers remained in this systematic review. There were 42 conference papers and 28 published papers. Two papers were excluded after the full-text reading for reasons number 4 and 5.

Here are some examples of excluded papers after the screening of the title and the abstract with exclusion explanations:

Here are two examples of why papers were excluded after full-text reading:

3.Household replenishment systems analysis

This section analyzes the selected papers based on the used approach and the implemented components. The 70 reviewed papers apply to the IoT technology or a combination of IoT and AI technologies, also referred to as Artificial Intelligence of Things (AIoT), and how those technologies can work with perishable item storage compartment systems.

Twenty-seven papers shed more light on the combination of AIoT components with perishable items storage compartments. Ten showed the results based on conceptual assumptions, 15 based on simulations, and 2 on the implementation of the system.

Forty-three papers drew the connection between IoT components and perishable item compartments. Twenty showed the solutions with conceptually assumed results, 17 of them were based on simulation, and six were on implementation of the system.

This section contains subsections that detail the IoT and AI components used in each reviewed paper. Each subsection will end with a table that summarizes the discussed matter. The component model and type will be written as mentioned in the reviewed papers. A checkmark (X) will be against the authors’ name if the type or model of the used component is included but not specified. Otherwise, it will be left blank or not listed in the subsections’ tables as it was not used.

This section ends with highlighted contributions, and limitations of and observations from each reviewed paper in Table 20 and Table 21, respectively. A complete table summarizes the subsections of the used components in Table 22. It forms the base for a data mining technique and a decision tree for a continuous variable decision. The frequency of used components among the reviewed papers is used to define the relationship between the used components and form them into categories. The continuous variable decision is used for the variables that depend on the outcome of each other, which applies to this paper. Therefore, this type of decision tree method is used, as shown in Table 23.

4.Findings and discussion

After analyzing the systems introduced in the 70 reviewed papers, this section will discuss the findings based on the descriptive analysis. Therefore, this section will have two sub-sections. The first section is about the used components and their functionalities, advantages, limitations, and disadvantages regarding the identification of stored items, as shown in Table 24. Finally, based on Table 22 and correlation analysis results Table 23, the decision tree will help develop a unified framework in which the used components are connected and communicate with each other to enhance the household management system performance.

4.2.A unified framework

Based on the previous analysis of the reviewed literature, this paper presents a unified framework in which all possible combinations of the mentioned components for the household replenishment system are presented. A decision-tree technique that uses a continuous decision variable, and is known as a regression tree, is used to connect and allow the components to communicate [54]. The unified framework is a showcase that gives researchers and manufacturers a starting point for enhancing the performance of the household replenishment system shown in Fig. 6. The components are categorized based on their functionalities, as shown in Table 25 followed by Table 26 for a description of how they work.

Fig. 4.

IoT vs. AIoT approach on smart storage compartments.

Therefore, the researchers and manufacturers would gain knowledge over the years about the components used in household replenishment systems based on the reviewed papers. For example, Loh et al. proposed a design of a system that can keep track of the free space inside the storage compartment. The authors divided the design into stages. Starting from the sensing sensors, through control and interface circuits, and ending with GSM circuit. The designed system was dedicated for space measurement and tracking food quantity [50]. Nayak et al. [62] and Panchal et al. [67] proposed a framework that presents specific components for remote monitoring. The framework was presented as a block diagram starting from the sensors ending with the market. The system was dedicated to monitor the stored items via IR sensor and initiate orders to the nearest market [62,67]. Hou et al. [35] proposed a framework that contains barcode and RFID technologies for food management. The proposed framework was split into several stages called units. The sensing, storage, control, push, and display units. The system was dedicated to the tagged items, and the non-tagged items can be entered manually [35]. Gürüler [30] proposed a block diagram that also shows specific components of the system. The proposed system was designed to allow the household to communicate with the storage compartment via SMS [30]. Hachani et al. [31], Floarea et al. [23], and Esmaeili [5] proposed a system architecture that uses RFID technology to capture stored items’ information. Kwon et al. proposed a system architecture that contains three stages. The first stage consists of capturing sensors. The second stage consists of database management. The final stage is the output stage [47]. Edward et al. [19] proposed a system architecture that consists of input, processing, connection medium, and output [19]. Wu et al. [89] proposed a system architecture that shows the identification of the items via Google Firebase using inside and outside cameras [89]. Anand et al. [6] have a slight difference. Anand et al. [6] added more sensors like gas, ultrasonic, and weighing sensors connected to a controller and to Google Firebase [6]. Nasir et al. [61] proposed a framework that shows three stages of the system: input, processing, and output. The proposed system was limited to specific components [61].

The unified framework presented in this paper in Fig. 6 is meant to be a generalized form of a framework derived from the specified frameworks presented in the reviewed papers. It is divided into four stages, and each stage consists of a combination of all components used in the reviewed papers. The starting triggers, input, processing, and output stages. The starting triggers stage consists of either automated or manual components that trigger the input stage components to start to work. The input stage’s components capture the data from the storage compartment and pass it through to the processing stage. The processing stage then processes the captured data into meaningful information and displays them accordingly via the output stage’s components. Later, the researchers and manufacturers could modify the components based on their perspectives and objectives. Figure 6 assigns a number to each component to help with components’ illustrations in Table 26.

However, the systems with the AIoT approach were developed in 2012 and 2013, then stopped until 2016, when they resumed growing. In 2020 and 2021, the use of AIoT approaches doubled the uses of IoT, Fig. 5, because of the technological revolution in Artificial Intelligence and human-less interactions with machines.

Fig. 5.

Application of IoT and AIoT over the years.

Fig. 6.

A unified framework for household replenishment system.

Based on literature, Table 25 classifies each component shown in Fig. 6 with its category. The categories are classified based on the components functionalities, as follows: 1) starting triggers are the components that trigger all input components to work except the barcode scanner and some RFID scanners that could work manually; 2) input components can be used for both quality management and quantity management, the quality management components are used to monitor the environment inside the storage compartment, the quantity management components are used to keep track of the stored items’ quantities and the household consumption habit, some of the quantity management components can identify the stored items and some do not; 3) the processing components receive the data from the input components to be processed and recorded; 4) then, the processed information can be retrieved by the output components to be displayed; 5) the processed information can be stored in either online or offline databases, and the connection medium that enables communication between all system components could vary.

Table 25

Categories of used components by the reviewed papers

Starting triggers	Input	Processing	Output	Storage and connectivity
User Commands	Gas Sensor	IoT Platform	Mobile App.	Offline database
Light Sensor	Climate Sensor	Controllers	Laptop	Online database
Door Sensor	IR Sensors	Desktop computer	Attached Tablet	Bluetooth
Timer	Ultrasonic	Attached Tablet	Desktop computer	Ethernet
	Weighing Sensor	Recognition Module		WiFi
	RFID System	eNose System		Internet Protocol
	Barcode Sensor			GSM Module
	Camera Module

Based on literature, a detailed illustration of the unified framework components shown in Fig. 6 and tied to their categories and associated with the assigned numbers is provided in Table 26.

Table 26

Detailed illustration of the framework components

Category	Component	Illustration
Starting Triggers Manually	(1-A) User Commands	The system receives user commands via GSM module (SMS) [50]
	(2-A) Voice Commands	The system can be woken up via household voice either from a smartphone or the built-in voice feature in the attached tablet (23-A) [31].
	(3-A) Pushbullet	A Pushbullet in the mobile application for the household to trigger the system to seek updates [61].
Starting Triggers Automated	4- Door open-close sensor	Once the storage compartment door is opened, the sensor triggers the input components to work [62].
	5- Light sensor	Senses when the light is turned on inside the storage compartment. It triggers the input components to work [18].
	6- Timer	A timer can be programmed to trigger the input components to work [9].
Input components Quality Management	7- Climate sensor	Monitors the humidity and the temperature inside the refrigerator. It could have a feature that controls the temperature based on household preferences [76].
	8- Gas sensor	Senses gas generated by organic food items such as fruit, vegetables, meat, etc., then updates the system [36].
Input components Quantity Management With out identification	9- Weight sensor	Measures and keeps track of the weight of stored item [36].
	10- IR sensor	Measures and keeps track of the distance between stored items and the untransparent liquid level [89].
	11- Ultrasonic sensor	Measures and keeps track of the distance between stored items and the transparent liquid level [6].
Input components Quantity Management With identification	12- Barcode reader	The household manually scans barcodes of the stored items while placing them in the storage compartment. (2-B) a built-in microphone in the attached tablet or (3-B) the microphone in the smartphone could be used to manually identify items by speaking their names while placing them in the storage compartment [24,34].
	13- RFID scanner	Updates and monitors the RFID-tagged items inside the storage compartment [44].
	14- Inside camera	Captures images of the stored items and sends them to the controllers to be identified and processed. In some cases, the images go directly to the household to monitor from a distance [11]. The inside camera is also used to track which food items have been taken by the household from inside the storage compartment to track the user consumption habit [80].
	15- Outside camera	The outside camera in the attached tablet (23-A) is used for facial recognition to recognize the household member opening the refrigerator and track the consumption habit [80].

Table 26

(Continued.)

Category	Component	Illustration
Processing components Controllers and Methods	16- Recognition module	Recognition module is a processing method that is an algorithm using artificial intelligence techniques. It processes images captured by (14) the inside camera, (15) the outside camera, natural language from (2-B) and (3-B) microphones, and the data collected by (20) the eNose system. The module can be located in (17) the controller, (18) a home desktop computer, and (19) an IoT platform [11,27,75,80].
	17- Controller	A controller is a device (e.g., Arduino) that receives data from all (input components), stores them in (21) the offline database or (22) the online database, and processes them. Then, it sends the processed data to the household. The household could reconfigure the controller via (18) the home desktop computer, (3-B) the smartphone mobile application, or (23-B) the GUI in the laptop [44,51,67].
	18- Desktop computer	The home desktop computer acts as an offline processor to collect data from all input components, store them in (21) the offline database, and process them. Then, it sends the processed data to the household. (23-A) The attached tablet works similarly. The household could edit the processed data directly [33].
	19- IoT platform	The IoT platform has several uses. One, it acts as (17) the controller and (18) the home desktop computer, but in a larger capacity and completely online. It uses (22) an online database to store and process the data. It enables the household to access information from a distance using (3-B) the smartphone mobile application or (23-B) the GUI in the laptop [13].
	20- eNose system	The eNose system collects the data from (8) gas sensors to send them to the controllers with an embedded (16) recognition module [27].
Output components	18- Desktop computer	The home desktop computer with a graphical user interface is also used locally as an output component that enables the household to view information about the system [27].
	(23-A) Attached tablet	The attached tablet is locally used via an installed mobile application to view information about the system [46].
	(3-B) Smartphone	Smartphone could be used via an installed mobile application to remotely view the system’s information or receive SMS messages via the GSM module [17].
	(23-B) Laptop	Laptop could be used as mobile device with a graphical user interface to remotely view information about the system [89].
Storage and Connectivity components	21- Local database	An offline database that could be used to store the processed data by (18) the home desktop computer or (17) the controller locally. It enables the household to manage it physically using (18) the home desktop computer [61].
	22- Online database	The processed data could be stored Online by (18) the home desktop computer or (17) the controller over the internet. The household can access the data locally through (18) the home desktop computer or from a distance using (3-B) the smartphone mobile application or (23-B) the GUI in the laptop [13].
	24- Internet	The Internet Protocols govern the connection between the controllers and the internet [34].
	(25-A) WiFi	Enables some components to communicate and connect to the internet [77].
	(25-B) Bluetooth	Enables some components to communicate [1].
	(25-C) Ethernet	Enables some components like (18) the home desktop computer to the internet [30].

5.Conclusion and future research directions

This paper focused on smart refrigerators as they always contain items that require monitoring. In contrast, other home appliances, such as washers and dryers, are used occasionally. However, other things could be learned from other smart appliances at home—for example, washer, dryer, coffee machine, etc. Therefore, in the future, there might be a need to conduct a review of other home appliances and show their effect under the smart home umbrella in general. This paper surveyed the relative work conducted on IoT and AI technologies applications on a home perishable items storage compartment to convert them into smart ones and show the households’ interaction. It focused on papers starting from 2000 using the mentioned keywords and criteria in methodology Section 2. A total of 70 papers were selected to be reviewed that applied either IoT or AIoT technologies toward smart refrigerators and/or cabinets following PRISMA search strategy. Twenty-seven papers shed more light on the combination of AIoT components with storage compartments for perishable items. Ten papers showed the results based on conceptual assumptions, fifteen papers based on simulations, and two papers on the implementation of the system. Forty-three papers drew the connection between IoT components and storage compartments for perishable items. Twenty papers showed the solutions with conceptually assumed results, seventeen of them were based on simulation, and six were on implementation of the system. These results indicate that the verification and validation of AIoT applications tend to be expensive. Therefore, this paper uses data mining techniques to build a continuous decision variable to analyze the reviewed papers that resulted in a unified framework including all possible combinations of used components. The used components were categorized into stages based on their functionalities. The starting triggers, input, processing, and output stages. A detailed description of the components used, functionalities, and limitations was provided.

The systematic review starts with section one, the introduction; section two, the research methodology; section three, the Household Replenishment Systems Analysis; section four, the findings and discussions; and ends with the conclusion and future directions section.

This comprehensive research pointed out the reviewed papers’ approaches, contributions, used components, and limitations.

The proposed framework was presented in this systematic review. It could be considered a showcase that gives researchers and manufacturers a starting point for enhancing the performance of the household replenishment system. It provides a visualization of all possible combinations of components used in literature for such a matter. Verification and validation to demonstrate the efficiency and effectiveness of the unified framework need to be addressed in the future.

Several summarized tables of the conducted review were provided. The summarized tables are intended to give researchers and manufacturers many factors to decide on the most effective combination of components that could be included or excluded with respect to their objectives. Moreover, the summarized tables could be used as guidelines to assign weight coefficients among the presented components based on their trendiness, frequency, functionalities, etc.

To the best of the authors’ knowledge, this would be the first comprehensive approach that includes all possible combinations of components used in household replenishment systems. Based on the thoroughness of the reviewed literature, the authors believe that this is a comprehensive framework expected to yield better results, but that is something that needs to be explored in other opportunities. In the spirit of continuous improvement, testing will result in outcomes that might lead us to make changes. More investigation needs to be done to include dissertations, theses, and books that tackle the same topic. Future work is to enlarge the scope and include introduced systems to enhance the household replenishment system to reduce food wastage in dissertations, theses, books, and patents. Furthermore, further investigation into smart systems and smart home appliances would be an introduction to extensive approaches like smart homes, markets, healthcare divisions, industries, and eventually smart cities.

Finally, this paper provides future research directions and sheds more light on areas of improvement for manufacturing companies gathered from the reviewed papers to enhance the household replenishment system. The proper use of IoT and AI technology may improve the household replenishment system in many ways:

Conflict of interest

None to report.