REFRIGERATOR AND CONTROLLING METHOD THEREOF

Abstract

A refrigerator includes: a body comprising a storage chamber; a door coupled to the body; a camera located on the body; and at least one processor operatively coupled to a memory and configured to execute one or more instructions stored in the memory, in which the one or more instructions cause the refrigerator to: based on detecting opening of the door, control the camera to capture the image of at least a part of the inside of the body and the door, identify, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition, and based on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image.

Description

BACKGROUND

1. Field

The disclosure relates to a refrigerator and a controlling method thereof, and more particularly, to a refrigerator that can obtain an optimal image for a door bin included in the refrigerator, and a controlling method thereof.

2. Description of Related Art

In general, a refrigerator is a home appliance that includes a storage chamber storing groceries, and a cold air supply device supplying cold air to the storage chamber, and thus, keeps groceries fresh for a certain period of time.

In recent developments, a camera is provided on the body of the refrigerator, which may be used to photograph the inside of the body. Furthermore, the refrigerator obtains information on the food currently stored in the refrigerator based on the photographed image, and provides the obtained information on the food to a user.

However, for obtaining information on food located in a door bin of a refrigerator, there is a need to photograph an optimal image for the door bin.

SUMMARY

According to an aspect of the disclosure, a refrigerator includes: a body including a storage chamber; a door rotatably coupled to the body to open or close the storage chamber, the door including a door bin; a camera on the body, the camera being configured to capture an image of an inside of the body and an inside of the door; a memory configured to store one or more instructions; and at least one processor operatively coupled to the memory and configured to execute the one or more instructions stored in the memory, wherein the one or more instructions, when executed by the at least one processor, cause the refrigerator to: based on detecting opening of the door, control the camera to capture the image of at least a part of the inside of the body and the door, identify, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition, and based on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image.

The camera may be on an upper end of the body, the first area of the inside of the body may be an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and the first line may be on an upper end of the door.

The one or more instructions, when executed by the at least one processor, further cause the refrigerator to: obtain a template image while the door is open and is halted during a specific time, compare the template image with the image captured by the camera, and identify, based on a result of the comparison of the template image with the image captured by the camera, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

The one or more instructions, when executed by the at least one processor, further cause the refrigerator to: detect a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera, determine, based on the plurality of images, whether the door is in a moving state, based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determine that the door is in a state of being open and halted, and based on determining that the door is in the state of being open and halted during the specific time, obtain, as the template image, an image at a time point in the state in which the door is open and halted.

The one or more instructions, when executed by the at least one processor, further cause the refrigerator to, based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the photographed image is a third predetermined value, identify that the predetermined condition is satisfied.

The refrigerator may further include an output device, and the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on receiving a user input to output the optimal door bin image, control the output device to output the optimal door bin image.

The door may further include a first door and a second door, and the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on detecting one of the first door or the second door is opened, control the camera to obtain an optimal door bin image corresponding to the one of the first door or the second door, and control the output device to output the optimal door bin image corresponding to the one of the first door or the second door.

The camera may be a wide angle camera, and the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: correct an image captured by the wide angle camera, and identify whether the first area of the inside of the body included in the corrected image and the first line included in the door included in the corrected image satisfy the predetermined condition.

The refrigerator may further include an output device, and the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline, obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image and cropping the area of interest, correct the image of the area of interest by applying a perspective transformation, and based on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest.

The one or more instructions, when executed by the at least one processor, further cause the refrigerator to: obtain, based on the image of the area of interest, information on incoming groceries or outgoing groceries, and control the output device to provide information on the incoming groceries or the outgoing groceries.

According to an aspect of the disclosure, a controlling method of a refrigerator including a body including a storage chamber, a door including a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera on the body, includes: based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door; identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; and based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.

The camera may be located on an upper end of the body, the first area of the inside of the body may be an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and the first line may be on an upper end of the door.

The identifying whether the image satisfies the predetermined condition may include: obtaining a template image while the door is open and is halted during a specific time; comparing the template image with the image captured by the camera; and identifying, based on a result of the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

The obtaining the template image may include: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera; determining, based on the plurality of images, whether the door is in a moving state; based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.

The determining whether the predetermined condition is satisfied may include, based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identifying that the predetermined condition is satisfied.

According to an aspect of the disclosure, a non-transitory computer readable medium has instructions stored therein which when executed by a processor of a refrigerator including a body including a storage chamber, a door including a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera located on the body, cause the processor to execute a method including: based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door; identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; and based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.

The camera may be on an upper end of the body, the first area of the inside of the body may be an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and the first line may be on an upper end of the door.

The identifying whether the image satisfies the predetermined condition may include: obtaining a template image while the door is open and is halted during a specific time; comparing the template image with the image captured by the camera; and identifying, based on a result the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

The obtaining the template image may include: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera; determining, based on the plurality of images, whether the door is in a moving state; based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.

The identifying whether the predetermined condition is satisfied may include, based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, determining that the predetermined condition is satisfied.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to one or more embodiments of the disclosure;

FIG. 2A to FIG. 2E are diagrams for illustrating an internal configuration of a refrigerator according to one or more embodiments of the disclosure;

FIG. 3 is a flow chart for illustrating a method for a refrigerator to obtain an optimal door bin image according to one or more embodiments of the disclosure;

FIG. 4 is a diagram for illustrating a first area of the inside of a door and a first line included in a door according to one or more embodiments of the disclosure;

FIG. 5A and FIG. 5B are diagrams for illustrating a method of monitoring a movement of a door of a refrigerator according to one or more embodiments of the disclosure;

FIG. 6A and FIG. 6B are diagrams for illustrating a method of detecting a state wherein a door of a refrigerator is completely open according to one or more embodiments of the disclosure;

FIG. 7 is a diagram for illustrating a predetermined condition for obtaining an optimal door bin image according to one or more embodiments of the disclosure;

FIG. 8A and FIG. 8B are diagrams illustrating an optimal door bin image output by a refrigerator according to one or more embodiments of the disclosure;

FIG. 9A to FIG. 9C are diagrams for illustrating one or more embodiments of obtaining an optimal door bin image by correcting a photographed image according to one or more embodiments of the disclosure;

FIG. 10A and FIG. 10B are diagrams for illustrating a method of obtaining an optimal door bin image by using a trained neural network model according to one or more embodiments of the disclosure;

FIG. 11A to FIG. 11D are diagrams for illustrating a method of correcting a photographed optimal door bin image and providing the image to a user according to one or more embodiments of the disclosure;

FIG. 12A to FIG. 12C are diagrams for illustrating a method of removing a person's face from an optimal door bin image and providing the image according to one or more embodiments of the disclosure;

FIG. 13A to FIG. 13C are diagrams for illustrating a method of managing incoming and outgoing of groceries by using an optimal door bin image according to one or more embodiments of the disclosure; and

FIG. 14 is a flow chart for illustrating a controlling method of a refrigerator for obtaining an optimal door bin image according to one or more embodiments of the disclosure.

DETAILED DESCRIPTION

The various embodiments of the disclosure and the terms used in the embodiments are not for limiting the technological characteristics described in the disclosure to specific embodiments, but they should be interpreted to include various modifications, equivalents, or alternatives of the embodiments.

In one or more examples, with respect to the detailed description of the drawings, similar or related components may be designated by similar reference numerals.

In addition, a singular form of a noun corresponding to an item may include one or a plurality of the item, unless instructed differently in the context.

Further, in the disclosure, each of the phrases “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one of the items listed together with the corresponding phrase among the phrases, or all possible combinations thereof.

In one or more examples, the term “and/or” includes a combination of a plurality of related components described, or any one component among the plurality of related components described.

In addition, terms such as “first,” “second,” and the like may be used just to distinguish a component from another component, and are not intended to limit a component in another aspect (e.g., importance or order).

Further, terms such as ‘front surface,’ ‘rear surface,’ ‘top surface,’ ‘bottom surface,’ ‘side surface,’ ‘left side,’ ‘right side,’ ‘upper part,’ ‘lower part,’ and the like used in the disclosure were defined based on the drawings, and the shapes and locations of respective elements are not limited by these terms.

In one or more examples, terms such as “include” and “have” should be construed as designating that there are such characteristics, numbers, steps, operations, elements, components, or a combination thereof described in the disclosure, but not as excluding in advance the existence or possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components, or a combination thereof.

In one or more examples, in case it is mentioned that one element is “connected with,” “coupled with,” “supported by,” or “contacted with” another element not only includes a case wherein the elements are directly connected, coupled, supported, or contacted, but also a case wherein the elements are indirectly connected, coupled, supported, or contacted through a third element.

Also, the description in the disclosure that one element is “on top of” another element not only includes a case wherein the one element contacts the another element, but also a case wherein still another element exists between the two elements.

A refrigerator according to one or more embodiments may include a body.

“The body” may include an inner cabinet, an outer cabinet arranged on the outside of the inner cabinet, and a heat insulating material provided between the inner cabinet and the outer cabinet.

“The inner cabinet” may include at least one of a case, a plate, a panel, or a liner forming a storage chamber. The inner cabinet may be formed as one body, or may be formed as a plurality of plates are assembled. “The outer cabinet” may form the exterior of the body, and may be coupled to the outside of the inner cabinet such that a heat insulating material is arranged between the inner cabinet and the outer cabinet.

“The heat insulating material” may insulate the inside of the storage chamber and the outside of the storage chamber such that the temperature inside the storage chamber can be maintained as an appropriate set temperature without being influenced by the outside environment of the storage chamber. According to one or more embodiments, the heat insulating material may include a foam heat insulating material. The foam heat insulating material can be molded by injecting and foaming urethane foam which is a mixture of polyurethane and a foaming agent between the inner cabinet and the outer cabinet.

According to one or more embodiments, the heat insulating material may additionally include a vacuum heat insulating material other than the foam heat insulating material, or it may consist only of a vacuum heat insulating material instead of the foam heat insulating material. The vacuum heat insulating material may include a core material, and an outer covering material that houses the core material and seals the inside with vacuum or pressure close to vacuum. However, the heat insulating material is not limited to the aforementioned foam heat insulating material or vacuum heat insulating material, and may include various materials that can be used for heat insulation.

“The storage chamber” may include a space defined by an inner cabinet. The storage chamber may further include an inner cabinet defining the space corresponding to the storage chamber. In the storage chamber, various articles such as groceries, medicine, cosmetics, etc. may be stored, and the storage chamber may be formed such that at least one side is opened for putting in or taking out articles.

The refrigerator may include one or more storage chambers. When two or more storage chambers are formed in the refrigerator, each storage chamber may have a different use from each other, and may be maintained at a different temperature from each other. To maintain different temperatures in separate storage chambers, each storage chamber may be partitioned from each other by a partition wall including a heat insulating material.

The storage chamber may be provided to be maintained in an appropriate temperature range according to its use, and may include a refrigerating chamber, a freezing chamber, or a temperature changing chamber that are divided according to their uses and/or temperature ranges. The refrigerating chamber may be maintained at an appropriate temperature for keeping an article refrigerated, and the freezing chamber may be maintained at an appropriate temperature for keeping an article frozen. “Refrigeration” may mean cooling an article to be cold within a limit that the article is not frozen, and as an example, the refrigerating chamber may be maintained in a range between 0° C. and 7° C. “Freezing” may mean cooling an article to be frozen or to be maintained in a frozen state, and as an example, the freezing chamber may be maintained in a range between −20° C. and −1° C. The temperature changing chamber may be used as any one of the refrigerating chamber or the freezing chamber by the user's choice or regardless of it.

The storage chamber may be referred to as various names such as a vegetable chamber, a fresh chamber, a cooling chamber, and an ice making chamber, etc., other than names such as a refrigerating chamber, a freezing chamber, and a temperature changing chamber, etc., and the terms such as “a refrigerating chamber,” “a freezing chamber,” and “a temperature changing chamber,” etc., used below, should be understood as meaning including a storage chamber that has a use and a temperature range corresponding to them.

According to one or more embodiments, the refrigerator may include at least one door that is constituted to open or close the opened one side of the storage chamber. The door may be provided to open or close each of the one or more storage chambers, or may be provided such that one door opens or closes the plurality of chambers. The door may be rotatably or slidably installed on the front surface of the body.

“The door” may be constituted to seal the storage chamber when the door is closed. The door may include a heat insulating material like the body such that the door insulates the storage chamber when the door is closed.

According to one or more embodiments, the door may include an outer plate of the door that forms the front surface of the door, an inner plate of the door that forms the rear surface of the door and faces the storage chamber, an upper cap, a lower cap, and a door heat insulating material that is provided on an inside of the door.

On the rim of the inner plate of the door, a gasket that seals the storage chamber by adhering to the front surface of the body when the door is closed may be provided. The inner plate of the door may include a dyke that projects rearward such that a door bin that can keep articles is mounted. Here, the door bin may be referred to as a door basket.

According to one or more embodiments, the door may include a door body, and a front panel that is detachably coupled to the front side of the door body and forms the front surface of the door. The door body may include an outer plate of the door that forms the front surface of the door body, an inner plate of the door that forms the rear surface of the door body and faces the storage chamber, an upper cap, a lower cap, and a door heat insulating material that is provided on the inside of the door.

The refrigerator may be divided into a French door type, a side-by-side type, a bottom mounted freezer (BMF), a top mounted freezer (TMF), or a one-door refrigerator, etc. according to the arrangement of the door and the storage chamber. A refrigerator with a French door type may include a first door and a second door.

According to one or more embodiments, the refrigerator may include a cold air supply device that is provided to supply cold air to the storage chamber.

“The cold air supply device” may include a machine, furniture, an electronic device, and/or a system that combined them that can generate cold air and guide the cold air, and can thereby cool the storage chamber.

According to one or more embodiments, the cold air supply device may generate cold air through a refrigerating cycle including compression, condensation, expansion, and evaporation processes of a refrigerant. To generate the cold air, the cold air supply device may include a refrigerating cycle device that has a compressor, a condenser, an expansion device, and an evaporator, which can drive a refrigerating cycle. According to one or more embodiments, the cold air supply device may include a semiconductor such as a thermoelectric element. The thermoelectric element may cool the storage chamber by heat generation and cooling operations through the Peltier effect, where electrical current flowing through a junction connecting two materials will emit or absorb heat per unit time at the junction to balance the difference in a chemical potential of the two materials.

According to one or more embodiments, the refrigerator may include a machine chamber that is provided such that at least some components belonging to the cold air supply device are arranged.

“The machine chamber” may be provided to be partitioned and insulated from the storage chamber to prevent transmission of heat generated from the components arranged in the machine chamber to the storage chamber. For radiating heat to the components arranged inside the machine chamber, the inside of the machine chamber may be constituted to be in communication with the outside of the body.

According to one or more embodiments, the refrigerator may include a dispenser that is provided on the door to provide water and/or ice. The dispenser may be provided on the door such that the user can access the dispenser without opening the door.

According to one or more embodiments, the refrigerator may include an ice making device that is provided to generate ice. The ice making device may include an ice making tray that stores water, an ice moving device that separates ice from the ice making tray, and an ice bucket that stores ice generated at the ice making tray.

According to one or more embodiments, the refrigerator may include a controller for controlling the refrigerator.

“The controller” may include a memory that stores or memorizes programs and/or data for controlling the refrigerator, and a processor that outputs a control signal for controlling the cold air supply device, etc. according to the programs and/or the data memorized in the memory.

The memory stores or records various kinds of information, data, instructions, programs, etc. necessary for the operations of the refrigerator. The memory may memorize temporary data that is generated while generating a control signal for controlling the components included in the refrigerator. The memory may include at least one of a volatile memory or a non-volatile memory or a combination of them.

The processor controls the overall operations of the refrigerator. The processor may execute the programs stored in the memory, and thereby control the components of the refrigerator. The processor may include a separate NPU that executes operations of an artificial intelligence model. Also, the processor may include a central processing unit (CPU), a graphics-dedicated processor (GPU), etc. The processor may generate a control signal for controlling the operations of the cold air supply device. For example, the processor may receive temperature information of the storage chamber from a temperature sensor, and generate a cooling control signal for controlling the operations of the cold air supply device based on the temperature information of the storage chamber.

Also, the processor may process a user input of a user interface according to the programs and/or the data memorized/stored in the memory, and control the operations of the user interface. The user interface may be provided by using an input interface and an output interface. The processor may receive a user input from the user interface. Also, the processor may transmit a display control signal for displaying an image on the user interface and image data to the user interface in response to a user input.

The processor and the memory may be provided integrally, or provided separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub-processor. The memory may include one or more memories.

According to one or more embodiments, the refrigerator may include a processor and a memory controlling all of the components included in the refrigerator, and include a plurality of processors and a plurality of memories that individually control the components of the refrigerator. For example, the refrigerator may include a processor and a memory that control the operations of the cold air supply device according to an output of the temperature sensor. Also, the processor may separately include a processor and a memory that control the operations of the user interface according to a user input.

The communication module may communicate with an external device such as a server, a mobile device, and other home appliances, etc. through an ambient access point (AP). The access point (AP) may connect the local network (LAN) to which the refrigerator or the user device is connected to a wide area network (WAN) to which the server is connected. The refrigerator or the user device may be connected to the server through the wide area network (WAN).

The input interface may include a key, a touch screen, a microphone, etc. The input interface may receive a user input and transmit it to the processor.

The output interface may include a display, a speaker, etc. The output interface may output various kinds of notifications, messages, information, etc. generated in the processor.

An optimal door bin image according to one or more embodiments of the disclosure is an image that photographed the door bin of the door while the door is open to satisfy a predetermined condition, and may be an image that satisfies an optimal condition for identifying information on groceries included in the door bin.

In one or more examples, a template image according to one or more embodiments of the disclosure may be an image that is photographed while the door is completely open to a set angle.

From below, the refrigerator according to various embodiments will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to one or more embodiments of the disclosure. As illustrated in FIG. 1, the refrigerator 100 may include a camera 110, an outputter 120, a communication interface 130, a microphone 140, a sensor 150, a memory 160, and a processor 170. The refrigerator 100 may be a device for keeping groceries or medicine, etc. at a predetermined temperature such that they are cooled to prevent the groceries from becoming spoiled. The refrigerator 100 according to one or more embodiments of the disclosure is illustrated in the form of a general refrigerator for homes, but is not limited thereto, and it may be a kimchi refrigerator, a beverage refrigerator, a cosmetics refrigerator, a freezer, etc.

The camera 110 is a component for photographing a subject and generating a photographed image, and here, the photographed image may include both of a moving image and a still image. “An image” in the disclosure may be a concept that includes both of an image output on the display 123 and an image frame photographed by the camera 110. In one or more examples, the photographed image may be referred to as an image captured by the camera 110.

Referring to FIGS. 1 and 2A, the camera 110 may photograph the areas of the storage chamber inside the body 230 of the refrigerator 100 and the door bins (or the door baskets, the pantries) of doors 210 and 220. The camera 110 may be provided in the upper end area inside the body 230 for photographing at least a part of the inside of the body 230 and the door bin areas of the doors 210 and 220. Also, the camera 110 may be provided in the side surface area or the lower end area inside the body 230 for photographing at least a part of the inside of the body 230. In addition, the camera 110 may be provided on the outside of the refrigerator 100 and photograph the outside of the refrigerator 100. For example, the camera 110 may be implemented not only as one camera, but also be implemented as a plurality of cameras depending on embodiments. In this regard, more detailed explanation will be described with reference to FIG. 2A to FIG. 2E.

FIG. 2A is a diagram illustrating a refrigerator when a door is opened according to one or more embodiments of the disclosure. The refrigerator 100 in FIG. 2A is illustrated such that a plurality of doors 210 and 220 are provided on both sides in the upper part. However, the disclosure is not limited thereto, and the refrigerator 100 may be implemented as a French door type, a side-by-side type, etc. according to the arrangement of the doors 210 and 220, the storage chamber 230, etc.

The refrigerator 100 may include the doors 210 and 220 on both sides in the upper part. Also, the refrigerator 100 may include a display 123 on one or more doors from among the doors 210, 220 on both sides in the upper part. Here, on the display 123, not only information on groceries stored in the refrigerator 100, but also various kinds of information (e.g., event information, notification information, recipe information, etc. received from the outside) may be provided.

As illustrated in FIG. 2A, the refrigerator 100 may further include the storage chamber 230. The storage chamber 230 may be accessible to one or more users by opening the doors 210 and 220, and store water, beverages, and refrigerated or frozen groceries. Here, the storage chamber 230 may include a plurality of accommodation spaces and storage spaces. The storage chamber 230 may be partitioned by partitions arranged inside the body. The storage chamber may be divided into a freezing chamber arranged in the lower part of the refrigerator 100 and a refrigerating chamber arranged in the upper part. However, the arrangement of the freezing chamber and the refrigerating chamber is not limited thereto, and they may be arranged while their locations are changed with each other.

The doors 210 and 220 may rotate by an angle set by a hinge (e.g., smaller than or equal to 300°), and open or close a part of the front surface of the storage chamber 230.

• • the second door 220 from among the plurality of doors 210, and 220 may include a display 123 that displays the functions and the setting of the refrigerator 100, and can be changed by the user's input (e.g., a touch or a selection of a button) on its surface. In one or more examples, a door from among the plurality of doors 210 and 220 may further include a dispenser that provides water, ice, or sparkling water and/or a handle that can be gripped, etc.

In one or more examples, as illustrated in FIG. 2A, the refrigerator 100 may include a camera 110 in the upper end area of the body for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220. Although FIG. 2A illustrates the camera 110 as being positioned in a center of the upper end area of the body, this configuration is merely an example. As understood by one of ordinary skill in the art, the camera 110 may be positioned at any desired location of the body.

FIG. 2B is an upward perspective view (e.g., top view) of the refrigerator 100, and as illustrated in FIG. 2B, the camera 110 may be arranged in the center of the upper end area of the body (e.g., the top table) for photographing all of the door bins of the first door 210 and the second door 220. FIG. 2C is a cross-sectional view that cuts the diagram in FIG. 2B by A-A′. As illustrated in FIG. 2C, the camera 110 may be arranged to be toward a downward direction by a predetermined angle (e.g., 30 degrees) for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220.

FIG. 2D and FIG. 2E are upward perspective views of the refrigerator 100, and are diagrams that illustrate the photographing ranges 240-1 and 240-2 in which the camera 110 performs photographing. FIG. 2D is a diagram that illustrates the photographing range 240-1 in which the camera 110 performs photographing while the doors 210 and 220 of the refrigerator 100 are completely open, and FIG. 2E is a diagram that illustrates the photographing range 240-2 in which the camera 110 performs photographing while the doors 210, 220 of the refrigerator 100 satisfy a predetermined condition. In one or more examples, the predetermined condition is a condition for obtaining an optimal door bin image, and in this regard, more detailed explanation will be described with reference to the drawings later.

In one or more examples, the camera 110 provides an image to the processor 170 as an optimal door bin image.

In one or more examples, the camera 110 may be implemented as a wide angle camera for photographing a wide view angle. However, the camera 110 may be any suitable camera known to one of ordinary skill in the art.

The outputter or output device 120 may provide various feedbacks. In particular, as illustrated in FIG. 1, the outputter or output device 120 may include a speaker 121, light emitting diodes (LED) 122, a display 123, etc., but this is merely an example, and the outputter output device 120 may further include another outputter (e.g., a haptic providing device, etc.).

Here, the speaker 121 may be provided inside or outside the refrigerator 100, and provide various auditory feedbacks through audio. The LED 122 may be provided inside the storage chamber 230 inside the refrigerator 100 or the doors 210, 220, and provide various visual feedbacks through an indicator in a specific form (e.g., an arrow, etc.) and flickering, etc. The display 123 may be located in at least some areas of the plurality of doors 210, 220, and provide various visual feedbacks to the user.

In one or more examples, the outputter 120 may output information guiding that groceries have been put into or taken out from the refrigerator, information on the groceries that have been put in or taken out, etc.

The communication interface 130 may perform communication with an external server or an external terminal device. In one or more examples, the communication interface 130 may transmit an image including groceries to an external server for obtaining information on the groceries, and receive information on the groceries from the external server. In one or more examples, the communication interface 130 may transmit information on groceries and information on the storage locations of the groceries to a user terminal, and receive a control instruction from the user terminal. Here, the communication interface 130 may directly perform communication with the user terminal, but this is merely an example, and the communication interface 130 may perform communication with an external user terminal through the server.

In one or more examples, the communication interface 130 may perform communication with various kinds of external devices by using various wireless communication technologies or mobile communication technologies. As such, wireless communication technologies, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared Data Association (IrDA), or near field communication (NFC), etc. may be included, and as mobile communication technologies, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, etc. may be included.

In one or more examples, the microphone 140 is a component that obtains an audio signal and converts it to an electric signal, and may be provided inside or outside the refrigerator 100. In particular, the microphone 140 may receive an audio signal including a user voice. Here, in the user voice, information on incoming or outgoing (hereinafter referred to as “incoming/outgoing”), and information on groceries (e.g., the types of groceries, the expiration dates of groceries, etc.) may be included.

In one or more examples, the sensor 150 may detect an operational state of the refrigerator 100 (e.g., the power or the temperature), or an external environmental state (e.g., a user state), and generate an electric signal or a data value corresponding to the detected state. In particular, the processor 170 may respectively measure the temperatures of the plurality of storage chambers of the refrigerator 100 through sensing values obtained through the sensor 150. In one or more examples, the processor 170 may recognize that the user is approaching through the sensor 150, and control the camera 110 to be in a preparation state. In one or more examples, the processor 170 may detect opening of the door through the sensor 150, and drive the camera 110.

The memory 160 may store an operating system (OS) for controlling the overall operations of the components of the refrigerator 100, and instructions or data related to the components of the refrigerator 100. In particular, the memory 160 may store various components for obtaining an optimal door bin image. In one or more examples, the memory 160 may store a database that stores information on the groceries stored in the refrigerator 100 (e.g., the type of the groceries, the amount of the groceries, the expiration dates of the groceries, the storage locations of the groceries, etc.).

In one or more examples, according to one or more embodiments, the memory 160 may store a machine learning model for recognizing groceries that were put into or taken out from the refrigerator 100 (e.g., a neural network model for object recognition). In one or more examples, the memory 160 may store a neural network model for obtaining the angles at which the doors 210, 220 included in the refrigerator 100 are opened by inputting an image photographed through the camera 110.

In one or more examples, the memory 160 may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory), a volatile memory (a memory included in the at least one processor 170 may be included), etc.

The processor 170 may control the refrigerator 100 according to at least one instruction stored in the memory 160.

In one or more examples, the processor 170 may include one or more processors. Specifically, the one or more processors may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or a random combination of the other components of the electronic device, and perform an operation related to communication or data processing. In one or more examples, the one or more processors may execute one or more programs or instructions stored in the memory. For example, the one or more processors may perform the method according to one or more embodiments of the disclosure by executing one or more instructions stored in the memory.

When the method according to one or more embodiments of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor, or may be performed by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by the method according to one or more embodiments, all of the first operation, the second operation, and the third operation may be performed by a first processor, or the first operation and the second operation may be performed by the first processor (e.g., a generic-purpose processor), and the third operation may be performed by a second processor (e.g., an artificial intelligence-dedicated processor). For example, an operation for obtaining an optimal door bin image may be performed through a generic-purpose processor such as a CPU, etc. which is a generic-purpose processor, and an operation of recognizing an angle at which the door is open or an operation for recognizing groceries by using a neural network model may be performed by an artificial intelligence-dedicated processor such as an NPU, etc.

The one or more processors may be implemented as a single core processor including one core, or it may be implemented as one or more multicore processors including a plurality of cores (e.g., multicores of the same kind or multicores of different kinds). When the one or more processors are implemented as multicore processors, each of the plurality of cores included in the multicore processors may include an internal memory of the processor such as a cache memory, an on-chip memory, etc., and a common cache shared by the plurality of cores may be included in the multicore processors. In one or more examples, each of the plurality of cores (or some of the plurality of cores) included in the multicore processors may independently read a program instruction for implementing the method according to one or more embodiments of the disclosure and perform the instruction, or the plurality of entire cores (or some of the cores) may be linked with one another, and read a program instruction for implementing the method according to one or more embodiments of the disclosure and perform the instruction.

The method according to one or more embodiments of the disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in the multicore processors, or they may be implemented by the plurality of cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, all of the first operation, the second operation, and the third operation may be performed by a first core included in the multicore processors, or the first operation and the second operation may be performed by the first core included in the multicore processors, and the third operation may be performed by a second core included in the multicore processors.

In the embodiments of the disclosure, the processor may correspond to a system on chip (SoC) wherein one or more processors and other electronic components are integrated, a single core processor, a multicore processor, or a core included in the single core processor or the multicore processor. Here, the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, etc., but the embodiments of the disclosure are not limited thereto.

In one or more examples, when opening of the doors 210 and 220 is detected, the processor 170 photographs at least a part of the inside of the body and the doors 210 and 220 by controller the camera 110. For example, the processor 170 identifies (or determines) whether at least one of the first area of the inside of the body or the first lines included in the doors 210 and 220 included in the image photographed through the camera 110 satisfies the predetermined condition. Here, the first area of the inside of the body may be an area of the lower end and the side surface of the body covered by the doors 210 and 220 when the doors 210 and 220 are closed, and the first lines included in the doors 210 and 220 may be lines located on the upper ends of the doors. When it is determined that an image of at least one of the first area of the inside of the body or the first lines included inside the doors 210, 220 satisfies the predetermined condition, the processor 170 stores the image as an optimal door bin image. Here, the processor 170 may not only store the optimal door bin image, but also control the communication interface 130 to transmit the image to an external device.

According to one or more embodiments, the processor 170 may obtain a template image while the doors 210 and 220 are in a state of being completely open (or opened by a predetermined amount) and halted during a specific time. The processor 170 may compare the template image and the photographed image, and identify whether the first area of the inside of the body and the first lines included in the doors satisfy the predetermined condition.

According to one or more embodiments, the processor 170 may detect a change of the first area of the inside of the body and the first lines included in the doors included in a plurality of images photographed through the camera 110, and determine whether the doors 210 and 220 are in a moving state. If the change of the first area of the inside of the body and the first lines included in the doors is smaller than or equal to a predetermined value while the doors 210 and 220 are opened, the processor 170 may determine that the doors 210 and 220 are in a state of being completely open and halted. If it is determined that the doors 210 and 220 are in a state of being completely open and halted during a specific time, the processor 170 may obtain an image on the time point when the doors 210 and 220 are completely open and halted as a template image.

According to one or more embodiments, if a difference value between the first area included in the template image and the first area included in the photographed image is a first predetermined value, or the length of the first line included in the photographed image is greater than a second predetermined value, or a door angle obtained based on the first line included in the photographed image is a third predetermined value, the processor 170 may determine that the predetermined condition is satisfied.

According to one or more embodiments, if a user input for outputting an optimal door bin image is received, the processor 170 may control the outputter 120 to output the optimal door bin image.

According to one or more embodiments, if one of the first and second doors 210 and 220 is opened, the processor 170 may obtain an optimal door bin image corresponding to the opened door. The processor 170 may control the outputter 120 to obtain the optimal door bin image corresponding to the opened door.

According to one or more embodiments, if the camera 110 is a wide angle camera, the processor 170 may correct an image photographed through the wide angle camera. The processor 170 may identify whether the first area of the inside of the body and the first lines included in the doors 210 and 220 included in the corrected image satisfy the predetermined condition.

According to one or more embodiments, the processor 170 may rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline. The processor 170 may obtain an image of an area of interest by extracting an area of interest in the rotated optimal door bin image and cropping the area. The processor 170 may correct the image of the area of interest by applying perspective transformation. If a user input for outputting the optimal door bin image is received, the processor 170 may control the outputter 120 to output the corrected image of the area of interest.

According to one or more embodiments, the processor 170 may obtain information on whether incoming groceries (e.g., groceries being stored in the refrigerator) or outgoing groceries (e.g., groceries being retrieved from the refrigerator) was detected through the image of the area of interest. The processor 170 may control the outputter 120 to provide information on groceries including the information on the incoming and outgoing groceries was detected.

From below, the disclosure will be described in more detail with reference to the drawings.

FIG. 3 is a flow chart for illustrating a method for a refrigerator to obtain an optimal door bin image according to one or more embodiments of the disclosure.

First, the refrigerator 100 may detect opening of the doors in operation S310. Here, the refrigerator 100 may detect opening of the doors by using a sensor provided on at least one of the doors 210 and 220 of the refrigerator 100 or the body. According to one or more embodiments, the refrigerator 100 may correctly detect the opening and closing states of the doors of the refrigerator by using a hall sensor located on the lower surface of the upper door and a magnet located on the upper surface of the lower door. In one or more examples, a hall sensor may be any sensor incorporating one or more Hall elements, each of which produces a voltage proportional to one axial component of a magnetic field vector B using a Hall effect. Hall sensors may be used for proximity sensing, positioning, and speed detection.

The refrigerator 100 may drive (e.g., control) the camera 110 in operation S320. Here, driving the camera 110 is an operation of initiating an operation of the camera 110 to obtain an image, and may be expressed as activating the camera 110, or initiating the camera 110. In one or more examples, in the aforementioned embodiment, it was explained that the camera 110 is driven when opening of the doors is detected, but this is merely one or more embodiments, and the camera 110 may be driven when approach of the user is detected. When the camera 110 is driven, the refrigerator 100 may obtain an image in real time through the camera 110.

The refrigerator 100 may monitor movements of the doors in operation S330. For example, the refrigerator 100 may monitor movements of the doors through a change of an image obtained through the camera 110. In particular, the refrigerator 100 may determine whether the doors 210 and 220 are in a moving state by detecting a change of the first area of the inside of the body and the first lines included in the doors 210 and 220 included in a plurality of images photographed through the camera 110.

In one or more examples, the first area of the inside of the door may be at least one area from among the lower end and the side surface of the body covered by the doors 210 and 220 when the doors 210 and 220 are closed. In in one or more examples, due to the limitations in the location and the photographing area of the camera 110, a part of the flange but not the entire flange of the inner cabinet of the body (or the storage chamber 230) may be shown in a photographed image. For example, the first area of the inside of the body may include some areas of the flange of the inner cabinet of the body (or the storage chamber 230). In one or more examples, the first lines included in the doors 210 and 220 may be lines located on the upper ends of the doors 210 and 220. Here, due to the limitations in the location and the photographing area of the camera 110, parts of the inner cabinets of the doors 210 and 220, but not the entire inner cabinets of the doors 210 and 220 may be shown in an image. For example, the first lines included in the doors 210 and 220 may be partial lengths of the flange surfaces on the upper sides of the inner cabinets constituting the doors 210 and 220.

As an example, FIG. 4 may be an image photographed by the camera 110 of the refrigerator 100 according to one or more embodiments of the disclosure. Here, the first area 410 of the inside of the body may be a partial area of the inside of the body located on the lower end of the image, and the first lines 420 included in the doors 210 and 220 may be partial lengths of the upper side surfaces included in the doors located on the upper end of the image. According to the degree of opening of the refrigerator doors, the size of the first area 410 of the inside of the body and the lengths or the angles of the first lines 420 included in the doors 210 and 220 may be changed. In one or more examples, the size of the first area 410 of the inside of the body and the lengths or the angles of the first lines 420 included in the doors 210 and 220 may be used to determine the degree of opening of the refrigerator doors.

The refrigerator 100 may analyze an image photographed by the camera 110 and determine whether the door is in a moving state. For example, the refrigerator 100 may analyze whether the first area changed from images photographed on any two time points t+1, t, and determine whether the door is in a moving state. Here, the any two time points may be time points of when two consecutive images were photographed.

In one or more examples, the refrigerator 100 may determine whether the door is in a moving state by the following formula 1.

$\begin{array}{cc}\begin{array}{c}\mathrm{abs}\left(A\left(t+1\right)-A\left(t\right)\right)=\mathrm{Diff}\\ \mathrm{count}\left(\mathrm{Diff}>{\mathrm{threshold}}_1\right)>{\mathrm{threshold}}_2\end{array}& \left[\mathrm{Formula}1\right]\end{array}$

Here, A(t+1) may be information on the first area on the t+1 time point, and A(t) may be information on the first area on the t time point. Here, the information on the first area is a pixel value of a pixel included in the first area, and it may be, for example, one value among 0 to 255. Diff may correspond to an absolute value of a difference value between A(t+1) and A(t).

The threshold₁ and the threshold₂ may be predetermined first value and second value. In one or more examples, the feature that Diff exceeds the predetermined first value means that a pixel value of a pixel included in the first area changed, and the feature that the number of pixels of which Diff exceeded the predetermined first value exceeds the predetermined second value may mean that changes of pixel values occurred in a plurality of areas (or a plurality of pixels) in the first area.

In one or more examples, if the size of an area in which a difference in the information on each area (e.g., a pixel value) in the first area on the t+1 time point and the t time point exceeds the predetermined first value exceeds the predetermined second value, the refrigerator 100 may determine that the door is in a moving state. For example, if the number of pixels of which difference in the pixel values in the first area on the t+1 time point and the t time point exceeds the predetermined first value exceeds the predetermined second value, the refrigerator 100 may determine that the door is in a moving state.

As an example, as illustrated in FIG. 5A and FIG. 5B, the refrigerator 100 may obtain an image photographed on at a time point t and an image photographed at a time point t+1. In one or more examples, the refrigerator 100 may identify whether the size of an area 520 in which a difference between a pixel value included in the first area 510 on the image photographed at the time point t+1 and a pixel value included in the first area 510 on the image photographed at the time point t exceeded the predetermined first value exceeds the predetermined second value. If the size of the area 520 in which the difference in the pixel values exceeded the predetermined first value exceeds the predetermined second value, the refrigerator 100 may determine that the door is in an opening state.

In one or more examples, in the aforementioned embodiment, it may be determined whether the door is opening through a change of the size (or the number of pixels) of the area in which pixel values changed in the first area of the inside of the body. However, this is merely one or more embodiments, and it may be determined whether the door is opening through a change of the length of the first line included in the door. In one or more examples, the refrigerator may determine whether the door is opening by using both of a change of the size of the first area and a change of the length of the first line.

In one or more examples, in the aforementioned embodiment, it may be determined whether the door is opening through a change of the size (or the number of pixels) of the area wherein pixel values changed in the fixed first area of the inside of the body. However, this is merely one or more embodiments, and it may be determined whether the door is opening through a change of the size of the first area. For example, the first area may not be a fixed area, but may be an area corresponding to a specific area of the inside of the body of the refrigerator (e.g., the lower end area of the body), and it may be determined whether the door is opening by comparing the sizes of the first areas in images photographed at the time point t and the time point t+1.

The refrigerator 100 may identify whether the doors 210 and 220 are in a state of being completely open (or opened to a predetermined degree) and halted during a specific time in operation S340. In one or more examples, the refrigerator 100 may analyze whether the first area changed from images photographed on any two time points (t+1, t), and determine whether the doors are in a state of being completely open and halted during the specific time. In one or more examples, the any two time points may be time points when two images were consecutively photographed.

In one or more examples, the refrigerator 100 may determine whether the door is in a moving state by the following formula 2.

$\begin{array}{cc}\mathrm{count}\left(\mathrm{Diff}>{\mathrm{threshold}}_1\right)>{\mathrm{threshold}}_3& \left[\mathrm{Formula}2\right]\end{array}$

In one or more examples, A(t+1) may be information on the first area at the time point t+1, and A(t) may be information on the first area at the time point t. In one or more examples, the information on the first area is a pixel value of a pixel included in the first area, and it may be, for example, one value among 0 to 255. Diff may correspond to an absolute value of a difference value between A(t+1) and A(t).

The threshold₃ may be a predetermined third value. In one or more examples, the feature that the number of pixels of which Diff exceeded the predetermined first value is smaller than or equal to the predetermined third value may mean that changes of pixel values rarely occur in the first area. In one or more examples, the predetermined third value may be an identical value to the predetermined second value described above, but this is merely one or more embodiments, and it may be a value smaller than the predetermined second value.

In one or more examples, if the size of an area in which a difference in the information on each area (e.g., a pixel value) in the first area the time point t+1 and the time point t exceeds the predetermined first value is smaller than or equal to the predetermined third value, the refrigerator 100 may determine that the door is in a state of being completely open (or opened to a predetermined degree) and halted during the specific time. For example, if the number of pixels of which difference in the pixel values in the first area at the time point t+1 and the time point t exceeds the predetermined first value is smaller than or equal to the predetermined third value, the refrigerator 100 may determine that the door is in a state of being completely open and halted during the specific time.

As an example, as illustrated in FIG. 6A and FIG. 6B, the refrigerator 100 may obtain an image photographed at a time point t and an image photographed at a time point t+1. In one or more examples, the refrigerator 100 may identify whether the size of an area 620 in which a difference between a pixel value included in the first area 610 on the image photographed at the time point t+1 and a pixel value included in the first area 610 on the image photographed at the time point t exceeded the predetermined first value is smaller than or equal to the predetermined third value. If the size of the area 620 in which the difference in the pixel values exceeded the predetermined first value is smaller than or equal to the predetermined third value, the refrigerator 100 may determine that the door is in a halted state during the specific time.

In particular, when a movement is not detected during the specific time, the refrigerator 100 may determine that the door is in a state of being completely open and halted.

The refrigerator 100 may obtain a template image and store the image in operation S350. In one or more examples, if it is determined that the doors 210, 220 are in a state of having been maintained in a completely open state during a specific time, the refrigerator 100 may obtain an image on the time point when the doors 210, 220 are completely open (the time point 0) as a template image. In one or more examples, the refrigerator 100 may define information on the size of the first area included in the template image as A(t−0). In one or more examples, the first area included in the template image may be referred to as a template area, and the information on the size of the first area included in the template image may be defined as a template area value.

As an example, FIG. 6A may be a template image that was obtained by photographing at the time point 0 when the door is in a state of being completely open.

In one or more examples, in the aforementioned embodiment, it may be determined whether the door is completely open through a change of the size (or the number of pixels) of the area wherein pixel values changed in the first area of the inside of the body. However, this is merely an example, and it may be determined whether the door is completely open through a change of the length of the first line included in the door. In one or more examples, the refrigerator may determine whether the door is completely open by using both of a change of the size of the area wherein pixel values changed in the first area and a change of the length of the first line.

The refrigerator 100 may store a template image, a template area, and a template area value in the memory 160. (In one or more examples, a template image, a template area, and a template area value may be generally referred to as a template.) In one or more examples, the refrigerator 100 may obtain an optimal door bin image by using at least one of the template image, the template area, or the template area value. As an example, the refrigerator 100 may update the template whenever the doors 210 and 220 are opened and store the updated template. Whenever the doors are opened, there is a possibility that the environment of a photographed area (e.g., the ambient lighting, the ambient contamination, the shadow, etc.) may change, and thus, the accuracy of an optimal door bin image may be improved by updating the template whenever the doors are opened. For example, a more accurate optimal door bin image may be obtained by dynamically setting the template whenever an optimal door bin image is obtained. As an example, the template may be used whenever the doors are opened or closed after initially being stored in the memory 160. For example, after the template is obtained initially, the operations S330 to S350 may be omitted. As an example, the template may be updated in accordance with a predetermined period.

The refrigerator 100 may identify whether an image photographed by the camera 110 satisfies a predetermined condition in operation S360. In one or more examples, the predetermined condition may be a condition for obtaining an optimal door bin image, and may be a condition related to at least one of the first area or the first line. In one or more examples, referring to FIG. 7, the predetermined condition may be a condition for 1) the area value (A(t), 2) of the first area 710, 2) the length l of the first line 720, and 3) the angle è that is constituted by the first line 720 and a reference line (e.g., a horizontal line), as illustrated in FIG. 7.

Regarding the condition 1), when a difference value between the size of the first area included in the template image and the size of the first area 710 included in a photographed image is the predetermined first value, the refrigerator 100 may identify that the predetermined condition is satisfied. For example, the refrigerator 100 may identify whether the predetermined condition is satisfied based on the following formula 3.

$\begin{array}{cc}A\left(t-O\right)-A\left(t\right)\approx S\left(\mathrm{specific}\mathrm{value}\right)& \left[\mathrm{Formula}3\right]\end{array}$

Here, A(t−0) may be information on the first area (or a pixel value of the first area) included in the template image, and A(t) may be information on the first area 710 (or a pixel value) on any time point. If a difference value between the information on the first area included in the template image and the information on the first area 710 at any time point satisfies a specific value S, the refrigerator 100 may identify that the predetermined condition is satisfied. In one or more examples, the specific value may be a value including a setting error. For example, if the sizes (or the number of pixels) of areas wherein pixel values changed in the first areas included in the template image and the image photographed on any time point satisfy the specific value S, the refrigerator 100 may identify that the predetermined condition is satisfied.

Regarding the condition 2), when the length of the first line included in a photographed image is greater than the predetermined value, the refrigerator 100 may identify that the predetermined condition is satisfied. In one or more examples, the refrigerator 100 may identify whether the predetermined condition is satisfied based on the following formula 4.

$\begin{array}{cc}l>L\left(\mathrm{specific}\mathrm{value}\right)& \left[\mathrm{Formula}4\right]\end{array}$

In one or more examples, the parameter l may be the length of the first line 720 on any time point. For example, if the length l of the first line is greater than a specific value on any time point t, the refrigerator 100 may determine that the predetermined condition is satisfied.

Regarding the condition 3), when it is determined that a door angle obtained based on the first line 720 included in a photographed image is the predetermined third value, the refrigerator 100 may identify that the predetermined condition is satisfied. For example, the refrigerator 100 may identify whether the predetermined condition is satisfied based on the following formula 5.

$\begin{array}{cc}\theta \approx T\left(\mathrm{specific}\mathrm{value}\right)& \left[\mathrm{Formula}5\right]\end{array}$

Here, è may be an angle between the first line 720 on any time point and the reference line (e.g., a horizontal line). For example, if an angle between the first line 720 at any time point t and the reference line (e.g., a horizontal line) is a specific value, the refrigerator 100 may determine that the predetermined condition is satisfied. Here, the angle between the first line 720 and the reference line (e.g., a horizontal line) may be determined as a door angle, and it may be an acute angle.

According to one or more embodiments, the refrigerator 100 may determine that the predetermined condition is satisfied in case all of the condition 1), the condition 2), and the condition 3) are satisfied, but this is merely an example, and the refrigerator 100 may determine that the predetermined condition is satisfied in case at least one of the condition 1), the condition 2), or the condition 3) is satisfied.

The refrigerator 100 may obtain an optimal door bin image in operation S370. Specifically, when a first image satisfying the predetermined condition is obtained, the refrigerator 100 may obtain the first image as the optimal door bin image and store it in the memory 160. For example, the refrigerator 100 may determine the first image satisfying the predetermined condition as an image on an optimal time point for photographing the door bin, and store the image as the optimal door bin image in the memory 160. As an example, the refrigerator 100 may store an image as illustrated in FIG. 7 satisfying the predetermined condition as the optimal door bin image.

In the aforementioned embodiment, it was explained that an optimal door bin image for one door from among the plurality of doors 210 and 220 is obtained, but this is merely an example, and optimal door bin images for each of the plurality of doors 210 and 220 may be obtained. For example, the refrigerator 100 can obtain a first optimal door bin image corresponding to the left door 210 and a second optimal door bin image corresponding to the right door 220.

According to one or more embodiments, if a user input for outputting the optimal door bin image is received, the refrigerator 100 may output the optimal door bin image. In one or more examples, the user input for outputting the optimal door bin image may be a touch input that is input on the display 123 provided on the refrigerator 100 or a user input that is input on a user terminal device communicatively connected with the refrigerator 100, but is not limited thereto. According to another embodiment, while the doors of the refrigerator 100 are closed, the refrigerator 100 may output the optimal door bin image through the display 123 provided on the doors 210 and 220.

The refrigerator 100 may output at least one of the first optimal door bin image corresponding to the left door 210 or the second optimal door bin image corresponding to the right door 220. For example, as illustrated in FIG. 8A, the refrigerator 100 may output the first optimal door bin image corresponding to the left door 210 (e.g., left pocket) with illustrated area 810, and as illustrated in FIG. 8B, the refrigerator 10 may output the second optimal door bin image corresponding to the right door 220 (e.g., right pocket) with illustrated area 820. Here, the UI screen including the first or the second optimal door bin image may include information on the doors corresponding to each of the first and second optimal door bin images.

In one or more examples, when only one of the plurality of doors 210 and 220 included in the refrigerator 100 is opened, the refrigerator 100 may photograph only one of the first and second optimal door bin images. For example, when only the left door 210 is opened, the refrigerator 100 may obtain only the first optimal door bin image corresponding to the left door 210, and when only the right door 220 is opened, the refrigerator 100 may obtain only the second optimal door bin image corresponding to the right door 220. In one or more examples, the refrigerator 100 may output the photographed optimal door bin image on the display 123, or transmit the optimal door bin image to an external user terminal.

When the camera 110 is a wide angle camera for photographing a wide view angle, as illustrated in FIG. 9A, a phenomenon in which the corners and the edges of a photographed image are distorted may occur. Due to this distortion, a problem that the accuracy of an operation of obtaining an optimal door bin image may deteriorate may occur. As an example, the refrigerator 100 may correct an image photographed by the camera 110 (e.g., the wide angle camera), and identify whether the first area of the inside of the body and the first lines included in the doors 210 and 220 included in the corrected image satisfy the predetermined condition.

For example, when an operation of analyzing an image and monitoring a movement of the door, an operation of obtaining a template image, and an operation of obtaining an optimal door bin image are performed in the operations S330 to S370 explained in FIG. 3, the refrigerator 100 may correct a photographed image. In one or more examples, as illustrated in FIG. 9A, in an image photographed by the camera 110, a distortion phenomenon wherein the first area is divided into a plurality of areas 910-1 and 910-2 in curvilinear forms, and the first lines 920-1 and 920-2 become curved lines occur, and thus, a problem that the accuracy of determining the optimal door bin image deteriorates. Accordingly, as illustrated in FIG. 9B and FIG. 9C, the refrigerator 100 may perform the operations described above by correcting the photographed image. For example, the refrigerator 100 may correct the photographed image based on a pre-stored conversion function. The refrigerator 100 may perform the operations described above based on one first area and the first lines in rectilinear forms that were obtained through image correction. As an example, as illustrated in FIG. 9B, the refrigerator 100 may obtain a template image by determining whether the doors 210 and 220 are in a completely open state based on the first area 930 of the corrected image. In one or more examples, as an example, as illustrated in FIG. 9C, the refrigerator 100 may obtain an optimal door bin image by determining whether the first area 940 and the first lines 950-1 and 950-2 of the corrected image satisfy the predetermined condition.

In one or more examples, when the refrigerator 100 outputs an optimal door bin image, the refrigerator 100 may correct a photographed image and output the image through the display 123, or transmit the image to the user terminal device. Like this, by providing an image wherein distortion has been corrected to the user, effects that aesthetics is improved and the user's satisfaction is improved can be achieved.

According to one or more embodiments, the refrigerator 100 may recognize a door angle by using a trained neural network model. In one or more examples, the door angle may correspond to an angle at which the doors 210 and 220 are opened. The refrigerator 100 may recognize a door angle based on a feature detection model 1020 or an angle detection model 1060. For example, as illustrated in FIG. 10A, the refrigerator 100 may obtain a photographed image 1010. The refrigerator 100 may input the photographed image 1010 into the trained feature detection model 1020, and obtain information on a feature 1030 included in the image 1010. In one or more embodiments, the feature detection model 1020 may be a neural network model that was trained to input an image photographed by a refrigerator and extract a feature of a door included in the image. The refrigerator 100 may input the obtained information on the feature 1030 into a door angle calculation model 1040. Here, the refrigerator 100 may calculate a door angle based on the location of the feature 1030 through the door angle calculation model 1040. The refrigerator 100 may obtain information on 45 degrees (1050) which is a door angle through the door angle calculation model 1040. As another example, as illustrated in FIG. 10B, the refrigerator 100 may obtain the photographed image 1010. The refrigerator 100 may input the photographed image 1010 into the trained angle detection model 1060, and obtain information on the door angles of the doors 210, 220 included in the image 1010. In one or more examples, the angle detection model 1060 may be a neural network model trained to input an image photographed by a refrigerator and obtain an angle at which a door included in the image is open (e.g., a door angle). The refrigerator 100 may obtain information on 45 degrees (1070) which is a door angle through an output of the angle detection model 1060. The refrigerator 100 may obtain an optimal door bin image based on a door angle obtained as explained in FIG. 10A and FIG. 10B.

According to one or more embodiments, the refrigerator 100 may correct a photographed optimal door bin image and provide the image to the user. For example, the refrigerator 100 may extract an area of interest among areas included in the photographed optimal door bin image, and generate an image of the area of interest based on the area of interest, and provide the image to the user.

Specifically, as illustrated in FIG. 11A, the refrigerator 100 may obtain an optimal door bin image 1110. In one or more examples, the refrigerator 100 may obtain a plurality of optimal door bin images (e.g., optimal door bin images corresponding to the left door and the right door), but for convenience, explanation will be described regarding one optimal door bin image 1110.

The refrigerator 100 may rotate the optimal door bin image 1110 to be aligned with a horizontal line based on a conversion guideline 1130, and as illustrated in FIG. 11B, obtain a rotated optimal door bin image 1120. Here, the conversion guide line 1130 may be a line that guides such that the door bins of the doors 210, 220 are located on the rotated optimal door bin image, as illustrated in FIG. 11B. Specifically, the refrigerator 100 may extract a door bin area in the photographed optimal door bin image 1110, and rotate the optimal door bin image 1110 such that the extracted door bin area corresponds to the conversion guideline 1130, and thereby, obtain a rotated optimal door bin image 1120.

As illustrated in FIG. 11C, the refrigerator 100 may obtain an image of an area of interest 1140 by extracting an area of interest in the rotated optimal door bin image and cropping the area. In one or more examples, the refrigerator 100 may obtain the image of the area of interest 1140 by extracting an area of interest corresponding to the conversion guideline 1130 (e.g., an area including a door bin area), and cropping the extracted area of interest, and enlarging the cropped area of interest.

The refrigerator 100 may correct the image of the area of interest by applying a perspective transformation, and obtain a corrected image of the area of interest 1150 as illustrated in FIG. 11D. In one or more examples, the perspective transformation corresponds to converting a quadrangle into any quadrangle for expressing a sense of perspective. Here, the refrigerator 100 may correct the image of the area of interest by using a pre-stored perspective conversion matrix.

If a user input for outputting an optimal door bin image is received, the refrigerator 100 may output the corrected image of the area of interest 1150 as illustrated in FIG. 11D. The image may be output to a display on the refrigerator, or may be transmitted to a mobile device of a user for display.

In an optimal door bin image photographed by a wide angle camera, it may be difficult to identify groceries depending on the location of the door. However, as described above, by correcting an optimal door bin image to an image of an area of interest and providing the image, the user can identify information on groceries located on the door more easily regardless of the location of the door.

In an optimal door bin image photographed by the refrigerator 100, as illustrated in FIG. 12A, a person 1210 outside the refrigerator 100 may be photographed, or a person 1220 checking the inside of the refrigerator 100 near the refrigerator 100 may be photographed. According to one or more embodiments, the refrigerator 100 may provide an optimal door bin image by removing the people 1210 and 1220 who were unintentionally photographed. For example, the refrigerator 100 may detect the people 1210 and 1220 who were unintentionally photographed, and perform blur processing or de-identification processing on the detected people 1210 and 1220.

As an example, when a person 1210 outside the refrigerator 100 is photographed, the refrigerator 100 may detect the person 1210 by using a face detector as illustrated in FIG. 12B, and perform blur processing on the area 1230 where the person 1210 is detected, and thereby correct the optimal door bin image. As another example, the refrigerator 100 may extract an area of interest (e.g., a door bin area) in the optimal door bin image, and as illustrated in FIG. 12C, perform blur processing on areas 1240-1 and 1240-2 other than the extracted area of interest. In one or more examples, when a person 1220 checking the inside of the refrigerator 100 near the refrigerator 100 is photographed, the refrigerator 100 may perform blur processing or de-identification processing on the user's face detected by the face detector. Here, the de-identification processing may correspond to performing image processing such that the person 1220 cannot be identified.

According to one or more embodiments, the refrigerator 100 may obtain information on groceries of which incoming and outgoing were detected through an image of an area of interest, and provide information on groceries including the information on the groceries of which incoming/outgoing were detected. In one or more examples, the refrigerator 100 may photograph an optimal door bin image, and when storing the image in the database, the refrigerator 100 may additionally store photographing information together with the optimal door bin image. In one or more examples, the photographing information is environment information when the optimal door bin image is photographed, and may include information on the photographing time, information on the user who put in and took out the groceries at the time of photographing, the time that the user used the refrigerator (the time between when the door was opened and when the door was closed), and the history of incoming/outgoing of the groceries on the door bin. For example, when an optimal door bin image 1310 as illustrated in FIG. 13A is stored, the refrigerator 100 may obtain information on “photographed at 12:16 on Nov. 24, 2023, used by the user A (dad), closed after using for 1 minute and 38 seconds, milk taken out from the door, milk put in on the shelf” as the photographing information.

According to one or more embodiments, the refrigerator 100 may recognize groceries that were put into and taken out from the door bin, and update a grocery list stored in the database. In one or more examples, the refrigerator 100 may obtain an optimal door bin image 1310 as illustrated in FIG. 13A. Then, the refrigerator 100 may obtain information on a plurality of groceries 1320-1 to 1320-5 included in the door bin by using an object recognition model that can obtain information on groceries. In one or more examples, the information on the groceries may be the types of the groceries, the number of the groceries, the sizes of the groceries, the amount of the groceries, the product names of the groceries, etc., but is not limited thereto. The refrigerator 100 may update the grocery list 1330 as illustrated in FIG. 13C based on the obtained information on the groceries 1320-1 to 1320-5. That is, the refrigerator 100 may add the groceries that have been put in to the grocery list 1330 based on the obtained information on the groceries 1320-1 to 1320-5, and delete the groceries that have been taken out from the grocery list 1330. In one or more examples, the refrigerator 100 may provide information on the groceries by classifying the grocery list 1330 by each storage location. For example, as illustrated in FIG. 13A, in case information on groceries was obtained from the optimal door bin image 1310 corresponding to the right door 220, the refrigerator 100 may update the list of the groceries on the right door bin in the grocery list 1330.

According to one or more embodiments, the refrigerator 100 may manage the history of incoming/outgoing of groceries on the door bin by detecting an empty hands or a motion on the doors 210, 220. Here, the refrigerator 100 may update the grocery list 1330 based on the history of incoming/outgoing of groceries on the door bin. Specifically, the refrigerator 100 may detect incoming/outgoing of groceries on the door bin in real time through detection of the user's motion. Here, the refrigerator 100 may obtain information on groceries that are put in and taken out and information on from which step of the door bin groceries are put in and taken out through detection of a motion and detection of groceries. Then, the refrigerator 100 may update the grocery list 1330 based on the information on groceries that are put in and taken out and the information on from which step of the door bin groceries are put in and taken out.

FIG. 14 is a flow chart for illustrating a controlling method of a refrigerator for obtaining an optimal door bin image according to one or more embodiments of the disclosure.

The refrigerator 100 detects opening of the doors in operation S1410. In one or more examples, the electronic device 100 may detect opening of the doors by using various sensors such as a hall sensor, etc.

The refrigerator 100 photographs at least a part of the inside of the body and the doors 210 and 220 by driving the camera 110 in operation S1420. In one or more examples, the camera 110 may be provided in the upper end area of the body.

The refrigerator 100 identifies whether at least one of the first area of the inside of the body or the first lines included in the doors included in the image photographed through the camera satisfies the predetermined condition in operation S1430. Here, the first area of the inside of the body may be an area of the lower end and the side surface of the body covered by the doors 210 and 220 when the doors 210 and 220 are closed, and the first lines included in the doors 210 and 220 may be lines located on the upper ends of the doors 210 and 220.

According to one or more embodiments, the refrigerator 100 may obtain a template image while the doors 210 and 220 are in a state of being completely open and halted during a specific time. In one or more examples, the refrigerator 100 may determine whether the doors 210, 220 are in a moving state by detecting a change of the first area of the inside of the body and the fist lines included in the doors 210 and 220 included in a plurality of images photographed through the camera 110. If the change of the first area of the inside of the body and the first lines included in the doors 210 and 220 is smaller than or equal to a predetermined value while the doors 210 and 220 are opened, the refrigerator 100 may determine that the doors 210, 220 are in a state of being completely open and halted. In one or more examples, if it is determined that the doors 210 and 220 are in a state of being completely open and halted during the specific time, the refrigerator 100 may obtain an image on the time point when the doors 210 and 220 are completely open and halted as a template image.

Then, the refrigerator 100 may compare the template image with the photographed image, and identify whether the first area of the inside of the body and the fist lines included in the doors 210 and 220 satisfy a predetermined condition. In one or more examples, if it is determined that a difference value between the first area included in the template image and the first area included in the photographed image is a first predetermined value, or the length of the first line included in the photographed image is greater than a second predetermined value, or a door angle obtained based on the first line included in the photographed image is a third predetermined value, the refrigerator 100 may determine that the predetermined condition is satisfied.

The refrigerator 100 stores a first image in which the first area of the inside of the body and the first lines included in the doors 210 and 220 satisfy the predetermined condition as an optimal door bin image in operation S1440.

According to one or more embodiments, if a user input for outputting an optimal door bin image is received, the refrigerator 100 may output the optimal door bin image.

According to one or more embodiments, in case the door includes the first door and the second door, if one of the first and second doors is opened, the refrigerator 100 may obtain an optimal door bin image corresponding to the opened door. Then, the refrigerator 100 may output the optimal door bin image corresponding to the opened door.

According to one or more embodiments, the camera 110 is a wide angle camera, and the refrigerator 100 may correct an image photographed by the wide angle camera. Then, the refrigerator 100 may identify whether the first area of the inside of the body and the first lines included in the doors 210, 220 included in the corrected image satisfy the predetermined condition.

According to one or more embodiments, the refrigerator 100 may rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline. The refrigerator 100 may obtain an image of an area of interest by extracting an area of interest in the rotated optimal door bin image and cropping the area. The refrigerator 100 may correct the image of the area of interest by applying perspective transformation. If a user input for outputting the optimal door bin image is received, the refrigerator 100 may output the corrected image of the area of interest.

According to one or more embodiments, the refrigerator 100 may obtain information on groceries of which incoming or outgoing was detected through the image of the area of interest. The refrigerator 100 may provide information on groceries including the information on the groceries of which incoming or outgoing was detected.

In one or more examples, the processor 170 according to one or more embodiments of the disclosure performs control to process input data according to predefined operation rules or an artificial intelligence model stored in the memory 160. The predefined operation rules or the artificial intelligence model are characterized in that they are made through learning.

In one or more examples, being made through learning means that a learning algorithm is applied to a plurality of learning data, and predefined operation rules or an artificial intelligence model having desired characteristic are thereby made. Such learning may be performed in a device wherein artificial intelligence is performed itself according to the disclosure, or performed through a separate server and/or system.

The artificial intelligence model (e.g., the first and second object detection networks) may consist of a plurality of neural network layers. At least one layer has at least one weight value, and performs an operation of the layer through the operation result of the previous layer and at least one defined operation. As examples of neural networks, there are a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN) and deep Q-networks, and a Transformer, but the neural network in the disclosure is not limited to the aforementioned examples excluding specified cases.

A learning algorithm is a method of training a specific subject device by using a plurality of learning data and thereby making the specific subject device make a decision or make prediction by itself. As examples of learning algorithms, there are supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but learning algorithms in the disclosure are not limited to the aforementioned examples excluding specified cases.

In one or more examples, the method according to the various embodiments of the disclosure may be provided while being included in a computer program product. A computer program product refers to a product, and it can be traded between a seller and a buyer. A computer program product can be distributed in the form of a storage medium that is readable by machines (e.g., a compact disc read only memory (CD-ROM)), or distributed directly on-line (e.g., download or upload) through an application store (e.g., Play Store™), or between two user devices (e.g., smartphones). In the case of on-line distribution, at least a portion of a computer program product (e.g., a downloadable app) may be stored in a storage medium readable by machines such as the server of the manufacturer, the server of the application store, and the memory of the relay server at least temporarily, or may be generated temporarily.

In one or more examples, the method according to the various embodiments of the disclosure may be implemented as software including instructions stored in machine-readable storage media, which can be read by machines (e.g., computers). The machines refer to devices that call instructions stored in a storage medium, and can operate according to the called instructions, and the devices may include an electronic device according to the aforementioned embodiments (e.g., a refrigerator).

In one or more examples, a storage medium that is readable by machines may be provided in the form of a non-transitory storage medium. Here, the term ‘a non-transitory storage medium’ only means that the device is a tangible device, and does not include signals (e.g., electromagnetic waves), and the term does not distinguish a case wherein data is stored semi-permanently in a storage medium and a case wherein data is stored temporarily. For example, ‘a non-transitory storage medium’ may include a buffer wherein data is temporarily stored.

In case an instruction is executed by a processor, the processor may perform a function corresponding to the instruction by itself, or by using other components under its control. An instruction may include a code that is generated or executed by a compiler or an interpreter.

While example embodiments of the disclosure have been shown and described, the disclosure is not limited to the aforementioned specific embodiments, and it is apparent that various modifications may be made by those having ordinary skill in the technical field to which the disclosure belongs, without departing from the gist of the disclosure as claimed by the appended claims. Further, it is intended that such modifications are not to be interpreted independently from the technical idea or prospect of the disclosure.

Claims

1. A refrigerator comprising: a body comprising a storage chamber;a door rotatably coupled to the body to open or close the storage chamber, the door comprising a door bin;a camera on the body, the camera being configured to capture an image of an inside of the body and an inside of the door;a memory configured to store one or more instructions; andat least one processor operatively coupled to the memory and configured to execute the one or more instructions stored in the memory,wherein the one or more instructions, when executed by the at least one processor, cause the refrigerator to: based on detecting opening of the door, control the camera to capture the image of at least a part of the inside of the body and the door,identify, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition, andbased on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image.

2. The refrigerator of claim 1, wherein the camera is on an upper end of the body, wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, andwherein the first line is on an upper end of the door.

3. The refrigerator of claim 1, wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: obtain a template image while the door is open and is halted during a specific time, compare the template image with the image captured by the camera, andidentify, based on a result of the comparison of the template image with the image captured by the camera, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

4. The refrigerator of claim 3, wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: detect a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera, determine, based on the plurality of images, whether the door is in a moving state,based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determine that the door is in a state of being open and halted, andbased on determining that the door is in the state of being open and halted during the specific time, obtain, as the template image, an image at a time point in the state in which the door is open and halted.

5. The refrigerator of claim 3, wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identify that the predetermined condition is satisfied.

6. The refrigerator of claim 1, further comprising: an output device, wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on receiving a user input to output the optimal door bin image, control the output device to output the optimal door bin image.

7. The refrigerator of claim 6, wherein the door further comprises a first door and a second door, and wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on detecting one of the first door or the second door is opened, control the camera to obtain an optimal door bin image corresponding to the one of the first door or the second door, andcontrol the output device to output the optimal door bin image corresponding to the one of the first door or the second door.

8. The refrigerator of claim 1, wherein the camera is a wide angle camera, and wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: correct an image captured by the wide angle camera, andidentify whether the first area of the inside of the body included in the corrected image and the first line included in the door included in the corrected image satisfy the predetermined condition.

9. The refrigerator of claim 1, further comprising: an output device,wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline,obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image and cropping the area of interest,correct the image of the area of interest by applying a perspective transformation, andbased on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest.

10. The refrigerator of claim 9, wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: obtain, based on the image of the area of interest, information on incoming groceries or outgoing groceries, andcontrol the output device to provide information on the incoming groceries or the outgoing groceries.

11. A controlling method of a refrigerator comprising a body comprising a storage chamber, a door comprising a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera on the body, the controlling method comprising: based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door;identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; andbased on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.

12. The controlling method of claim 11, wherein the camera is located on an upper end of the body, wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, andwherein the first line is on an upper end of the door.

13. The controlling method of claim 11, wherein the identifying whether the image satisfies the predetermined condition comprises: obtaining a template image while the door is open and is halted during a specific time;comparing the template image with the image captured by the camera; andidentifying, based on a result of the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

14. The controlling method of claim 13, wherein the obtaining the template image comprises: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera;determining, based on the plurality of images, whether the door is in a moving state;based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; andbased on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.

15. The controlling method of claim 13, wherein the identifying whether the predetermined condition is satisfied comprises: based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identifying that the predetermined condition is satisfied.

16. A non-transitory computer readable medium having instructions stored therein which when executed by a processor of a refrigerator comprising a body comprising a storage chamber, a door comprising a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera located on the body, cause the processor to execute a method comprising: based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door;identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; andbased on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.

17. The non-transitory computer readable medium of claim 16, wherein the camera is on an upper end of the body, wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, andwherein the first line is on an upper end of the door.

18. The non-transitory computer readable medium of claim 16, wherein the identifying whether the image satisfies the predetermined condition comprises: obtaining a template image while the door is open and is halted during a specific time;comparing the template image with the image captured by the camera; andidentifying, based on a result the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.

19. The non-transitory computer readable medium of claim 18, wherein the obtaining the template image comprises: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera;determining, based on the plurality of images, whether the door is in a moving state;based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; andbased on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.

20. The non-transitory computer readable medium of claim 18, wherein the identifying whether the predetermined condition is satisfied comprises: based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, determining that the predetermined condition is satisfied.