Patent Application
20220222844
The present invention relates to a method for measuring food, and in particular, to a method of measuring the amount of food using volume.
In recent years, as more people want to maintain a healthy diet such as well-being and diet, the demand for food measuring technology is increasing.
Even in places where food is served to multiple people, such as schools, companies, the military, government offices, hospitals, by measuring the amount of food served and the amount of leftover through measurement of the amount of food served and distributed to people, there are many advantages such as being able to conduct efficient food distribution in anticipation of the amount of demand and supply and to manage the calories of those who are served of the food.
However, since most of the currently known technologies are merely searching for images taken through a camera, the accuracy is remarkably degraded, and there is a problem that a larger error occurs in subsequent steps such as calorie calculation since the accuracy in image search is poor.
Accordingly, the inventor has come up with an invention that can accurately analyze and calculate food, not just searching for food by image.
The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and it may therefore contain information that does not form the prior art that is already known to a person of ordinary skill in the art.
The present invention for solving the above-described problem is directed to providing a method for measuring food for calculating volume of food by using height information on each pixel of an image photographed through a stereo camera or a depth measuring device.
In addition, the present invention may provide a method for measuring food for correcting the calculated volume of food by using information of a serving tableware.
The technical problems to be solved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art from the following description.
The method for measuring food performed by a computer according to an embodiment of the present invention for solving the above-described problem includes steps of: receiving image data of tableware containing food and depth data of the tableware containing the food, when the tableware containing food is photographed by a photographing unit; calculating a tilt of the tableware image from the tableware image data, and correcting the image data of the tableware containing food and the depth data of the tableware containing food according to the calculated tilt; extracting a shape of the tableware using the corrected depth data of the tableware containing food, and based on this, extracting food image data accommodated in the tableware from the corrected tableware image data; determining a type of the food based on the extracted food image data; and calculating volume of the food based on the corrected depth data of the tableware containing food.
In addition, a server for measuring food according to an embodiment of the present invention for solving the above-described problem includes a receiving unit to receive image data of tableware containing food and depth data of the tableware containing the food, when the tableware containing food is photographed by a photographing unit; a correction unit to calculate a tilt of the tableware image from the tableware image data, and correcting the image data of the tableware containing food and the depth data of the tableware containing food according to the calculated tilt; an extraction unit to extract a shape of the tableware using the corrected depth data of the tableware containing food, and based on this, to extract food image data accommodated in the tableware from the corrected tableware image data; a determination unit to determine a type of the food based on the extracted food image data; and a calculation unit to calculate volume of the food based on the corrected depth data of the tableware containing food.
In addition, a device for measuring food according to an embodiment of the present invention for solving the above-described problem includes a receiving unit to receive image data of tableware containing food and depth data of the tableware containing the food, when the tableware containing food is photographed by a photographing unit; a correction unit to calculate a tilt of the tableware image from the tableware image data, and correcting the image data of the tableware containing food and the depth data of the tableware containing food according to the calculated tilt; an extraction unit to extract a shape of the tableware using the corrected depth data of the tableware containing food, and based on this, to extract food image data accommodated in the tableware from the corrected tableware image data; a determination unit to determine a type of the food based on the extracted food image data; and a calculation unit to calculate volume of the food based on the corrected depth data of the tableware containing food.
In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium for recording a computer program for executing the method may be further provided.
According to the present invention as described above, there is an effect of calculating accurate volume of food by calculating the volume of food by using height information on each pixel of an image photographed through a stereo camera or a depth measuring device.
In addition, the present invention has an effect of more accurately calculating volume of food by correcting the calculated volume of food by using information on size, depth, and capacity of spaces formed in the serving tableware.
The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.
The above and other aspects, features, and advantages of the present invention will become more apparent to those of ordinary skill in the art by describing embodiments thereof in detail with reference to the accompanying drawings, in which:
Advantages and features of the present invention, and a method of achieving them will be apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various forms. Rather, the description of the embodiments of the present invention is provided so that this disclosure will be thorough and complete and will fully convey the concept of the invention to those of ordinary skill in the art. Accordingly, the present invention is only defined by the scope of the appended claims.
The terminologies used in this specification are for the purpose of describing embodiments only and are not intended to be limiting to the present invention. In this specification, the singular form also includes the plural form unless specifically specified in the phrase. The terms “comprise” and/or “comprising,” when used in this specification, do not preclude the presence or addition of one or more other elements other than elements mentioned. Like reference numerals designate like elements throughout the specification, and the term “and/or” includes each and all combinations of one or more of the mentioned elements. Although “first”, “second”, and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one element from another element. Therefore, it goes without saying that the first element mentioned hereinafter may be the second element within the technical idea of the present invention.
Unless otherwise defined, all terms (including technical and scientific terms) used in this specification will be used as meanings that can be commonly understood by those of ordinary skill in the art. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly and specifically defined.
Hereinafter, embodiments of the present invention are described in detail with reference to the accompanying drawings.
Prior to the description, the meaning of terms used in this specification is briefly described. However, it should be noted that the description of terms is not intended to limit the technical idea of the present invention unless explicitly described as limiting the present invention, since it is intended to aid understanding of the present specification.
A method for measuring food according to an embodiment of the present invention will be described with reference to
The method for measuring food according to an embodiment of the present invention may be performed at a food serving section or a tableware return section of a restaurant, and the restaurant may be a self-service restaurant such as a cafeteria or a buffet, a place where a user moves a tableware 500 to receive food, and may be a place where meals are provided to a group such as a school, a company, or the military. Here, the restaurant may include a food serving section for receiving food or a tableware 500 and a tableware return section for throwing away the leftover and returning the tableware 500 after eating, but is not limited thereto.
However, the method for measuring food according to an embodiment of the present invention is not limited to being performed in a restaurant, particularly a self-service restaurant, and may be performed in other types of restaurants or at home, or may be performed at each seat.
In addition, for convenience of description, the description is made using the tableware 500 with a shape of a food tray divided into spaces 510, but the shape of the tableware 500 is not limited thereto.
Referring to
The receiving unit 110 normalizes the received image of tableware containing food, and the normalization refers to adjusting or transforming the received image of tableware containing food according to predetermined criteria (e.g., size, brightness, rotation, tilt, etc.).
If the photographing unit 250 can acquire image data and depth data, there is no limitation on the configuration thereof. For example, the photographing unit 250 may include at least one of an RGB camera, a 2D camera, a 3D camera, a Time of Flight (ToF) camera, a light field camera, a stereo camera, an event camera, and an infrared camera, and if it can measure image data and depth data, there is no limitation on the configuration thereof.
Meanwhile, in addition to photographing food or tableware containing food, the photographing unit 250 may photograph biometric information of a taker to identify the taker. Here, the biometric information of the taker may be the face, iris, or fingerprint of the taker, but is not limited thereto. One photographing unit 250 may simultaneously or separately photograph food or tableware containing food and biometric information of the taker, but the photographing unit may be divided into a photographing unit 250a for photographing food or tableware containing food and a photographing unit 250b for photographing biometric information of the taker.
In some embodiments, when photographing the tableware 500 moving along a conveyor belt 270, the photographing unit 250 of the device for measuring food 200 may acquire depth data in addition to image data, even if it includes a single camera. It is because if the single photographing unit 250 respectively photograph the moment when the tableware 500 is located at a first position on the conveyor belt 270 and the moment when the tableware 500 is moved and located at a second position on the conveyor belt 270, this may obtain the same effect as photographing the still tableware 500 using the photographing unit 250 including a plurality of cameras.
Since the camera is a device that recognizes light, it is highly affected by the environment such as light reflection. Therefore, it is necessary to minimize the influence of the environment by taking pictures using the photographing unit 250 including two or more cameras. However, even if photographing is performed using the photographing unit 250 including two or more cameras, since it may be affected by the environment, correction thereof is required.
Therefore, in order to optimize according to the characteristics and environment of the restaurant, the computer may control its learning module to learn the situation of the field through the images taken, and to recognize the environment condition based on filter processing, fixed objects and reference objects (background targets described later).
Meanwhile, when photographing the tableware 500 using the device for measuring food 200 according to an embodiment of the present invention, the device for measuring food 200 may detect identification information 530 on the tableware 500 or identification information on the taker as well as photographing the tableware 500 from the upper side using the photographing unit 250. That is, it is possible to identify the taker by using the device for measuring food 200 simultaneously or separately from the step of receiving the tableware image (S20) at step S21.
Specifically, referring to
For example, as shown in
In some embodiments, when the identification information of the dish 500 is located on the upper side of the tableware 500, the device for measuring food 200 may not include a separate identification information sensing unit 251, and may identify the identification information of the tableware 500 by analyzing a tableware image measured using the photographing unit 250.
Besides, identification information on the taker may be detected by an authentication unit 160, for example, by applying some methods such as facial recognition which identifies biometric information using image capture of the taker, iris recognition, or fingerprint recognition.
In addition, the authentication unit 160 may detect identification information on the taker by recognizing a tag including an RFID chip held by the taker or a tag including identification information in various code types such as a QR code, a barcode, a combination of numbers or letters, and a combination of figures and symbols.
In the case of using the method for measuring food according to an embodiment of the present invention, various information may be calculated using the measured result, and there may be a difference in information calculated depending on what information the device for measuring food 200 can recognize among information on the tableware 500 and information on the taker 530.
First, if both the device for measuring food 200 installed in the food serving section and the device for measuring food 200 installed in the tableware return section may recognize identification information 530 on the tableware 500, and may recognize only identification information on the taker, information on each taker and information on a group including the corresponding taker may be calculated using the method for measuring food according to the present embodiment.
In contrast, if both the device for measuring food 200 installed in the food serving section and the device for measuring food 200 installed in the tableware return section may not recognize the identification information on the taker, and may recognize only the identification information 530 on the tableware 500, the information on each taker may not be calculated, and only the information on a group including the corresponding taker may be calculated using the method for measuring food according to the present embodiment.
Besides, if the device for measuring food 200 installed in the food serving section may recognize both the identification information 530 on the tableware 500 and the identification information on the taker, and the device for measuring food 200 installed in the tableware return section may not recognize the identification information 530 on the tableware 500 but may recognize the information on the taker, both the information on each taker and the information on a group including the corresponding taker may be calculated using the method for measuring food according to the present embodiment; since even if the device for measuring food 200 installed in the tableware return section may not recognize the identification information 530 on the tableware 500, the identification information 530 on the tableware 500 held by a specific taker may be specified by using the information from the food serving section.
Meanwhile, a food identification device 30 may not include a separate authentication unit 160 and may receive input of identification information from the taker through an input unit (not shown) of the device for measuring food 200.
Therefore, in some embodiments, in the step of receiving, by the receiving unit 110, the tableware image taken through the photographing unit 250 (step S20), the receiving unit 110 may also receive at least one of identification information on the tableware 500 and identification information on the taker in addition to the tableware image.
Subsequently, referring to
More particularly, the correction unit 150 may remove a noise component using the image data of tableware containing food, recognize a tilt of the tableware image, and correct the image into a horizontal state image. The corrected image may be used in a step of calculating volume (S50) described later.
Since the tableware image may be photographed unclear or the tableware image may be photographed while the angle of the tableware is inclined, the correction unit 150 recognizes the shaking or tilt of the tableware image, select the tableware image with the least shaking or tilt, and correct the tableware image into a horizontal state image, which allows to accurately calculate the volume of the food.
First, the correction unit 150 may analyze a plurality of normalized tableware images in order to select a tableware image with the least shaking among the normalized tableware images. For example, the correction unit 150 may check the degree of blurring of the image in units of each pixel, and through this, select a tableware image that has the least shaking and is clear.
Next, the correction unit 150 may calculate a distance from the photographing unit 250 to a plurality of exposed areas for an area in which the tableware 500 is not covered by food and exposed in the tableware image, and may calculate a tilt of the tableware 50 from the tableware image using the calculated distances to the plurality of exposed areas. Next, the correction unit 150 may correct the photographed tableware image to be a horizontal state using the calculated tilt.
Specifically, the correction unit 150 may extract a shape of the tableware 500 using depth information from the depth of tableware image. Since the tableware 500 and the background other than the tableware 500 differ in depth information in the tableware image, using this, the correction unit 150 may extract the tableware 500 from the tableware image.
The correction unit 150 identifies flat areas among areas exposed without being covered by food in the tableware 500, select at least 3 or more points among the flat areas, and identifies depth information of the selected 3 or more points.
Using the 3 or more points, it may be possible to create a virtual plane and check a slope of the plane. That is, the correction unit 150 may calculate a tilt of the tableware 500 using depth information of the 3 or more selected points, and correct the photographed tableware image using the calculated tilt.
In some embodiments, after selecting at least 3 or more points in the flat areas, with respect to the 3 or more selected points the accuracy of the calculated tilt may be improved by using an average depth information of the surrounding points of each point as a depth information of each point and by using a relatively large amount of information to calculate the tilt.
In some embodiments, in the process of calculating the tilt of the tableware 500 using depth information of the selected 3 or more points, a gradient value according to a line connecting each point may be derived, through which the tilt of the tableware 500 may be calculated and the photographed tableware image may be corrected using the calculated tilt.
In some embodiments, the correction unit 150 may calculate the tilt of the tableware 50 from the photographed tableware image by identifying an outline of the tableware 50 from the tableware image and by comparing the outline of the tableware 50 from the photographed tableware image and the outline of the tableware 50 from the non-tilted tableware image that is for use as a reference. Next, the correction unit 150 may correct the photographed tableware image using the calculated tilt.
In addition, since noise may occur according to the situation of the field (shake, light reflection, etc.) in two image data of tableware containing food received through the receiving unit 110, the noise may be removed by comparing the two tableware images.
Next, referring to
Since the tableware image includes both the tableware 500 and the food image, the food image data may be extracted from the tableware image. For example, the extraction unit 120 may extract food image data using an artificial intelligence model learned to distinguish food and non-food items. Here, when the tableware 500 includes a plurality of spaces 510, the extraction unit 120 may extract food image data for each space 510.
Meanwhile, referring to
Next, referring to
The determination unit 130 may determine the food type using the extracted food image data. For example, the determination unit 130 may include an artificial intelligence model learned based on image in order to recognize the type of food, but the method of determining the type of food is not limited thereto.
In the method for measuring food according to the present embodiment, in order to increase the recognition rate of food types, the learned artificial intelligence model may be optimized for an individual target (e.g., an individual restaurant or an individual taker) for which the method for measuring food is used, in the following manner.
First, referring to
The learned model to be optimized is a master model, and the master model is a model that has been trained to classify types of food by using image data photographed from the device for measuring food 200 or the user terminal 300 or processed results as a data set.
Since the master model is constructed from all the captured image data or processed results used in the method for measuring food as a data set, in order to determine the type of specific food image data, the master model determines what kind of food the specific food image data is from dozens, hundreds, or thousands of food type classes.
Optimizing the master model is to reduce the number of cases for food type classes that need to be considered in order to determine the food type of specific food image data. To this end, it may be possible to obtain optimization information for individual target such as a restaurant or a taker.
That is, the receiving unit 110 may receive optimization information for an individual target. The optimization information may be, for example, information on something consumed by each individual target or something having a high probability of being consumed by a taker that uses an individual target such as menu information sold or provided by an individual restaurant or types of food frequently consumed by the individual taker.
Next, referring to
The determination unit 130 calculates the local model for an individual target by selecting a food type class that is highly related to the individual target based on the optimization information and optimizing the class of the master model in a manner of reducing a considered food type class from the master model to a food type class that is highly related to the individual target.
Next, referring to
The determination unit 130 may advance the local model by performing learning of the calculated local model using image data related to the individual target or a result processed in relation to the individual target.
Here, the image data related to the individual target may be image data photographed within an individual restaurant as an individual target or a result of processing image data photographed within the individual restaurant, or image data photographed by an individual taker as an individual target or a result of processing image data photographed by the individual taker.
Meanwhile, in the process of advancing the local model by learning the calculated local model by using image data related to an individual target or a result processed in relation to the individual target, the determination unit 130 may perform learning while changing various variables such as learning parameters and learning cycles in the process of performing learning using the artificial intelligence algorithm, and perform an advancement process using a learning model with the highest accuracy among the performed learning methods.
Next, referring to
In some embodiments, as a preceding step, a step of receiving, by the receiving unit 110, food list information provided by a restaurant (step S10) may be further included.
In this case, in the step of determining the food type (step S40), the determination unit 130 may determine the type of each food by matching the extracted food image data and food list information.
In this way, the determination unit 130 may independently determine the type of food by analyzing food image data, or may determine the type of food by matching with stored food list information, and the method of determining the type of food is not limited thereto.
Next, referring to
As mentioned above, since the tableware image was photographed through the photographing unit 250 capable of acquiring depth data, height information (3D distance data) for each pixel is included in the food image data as depth data, and so this may be used to calculate the volume of each food.
Here, the computer includes a database 170 in which tableware images and tableware information including sizes, depths and capacities of the plurality of spaces 510 formed in the tableware 500 are stored.
In addition, the step of calculating volume (step S50) may further include a step of correcting, by the correction unit 150, the calculated volume of each food using the tableware information including the size, depth and capacity of the space 510 in which each food is accommodated.
Since the depth of each space 510 is different according to the type of tableware 500, the correction unit 150 may determine more accurately how much food has been accumulated by using the tableware information. Since the database 170 of the computer according to the present embodiment stores data on the tableware 500 used in a restaurant, the volume of each food extracted using tableware information may be more accurately calculated by using the method for measuring food according to the present embodiment.
For example, when the determination unit 130 determines that the type of food accommodated in a specific space 510 in the tableware 500 is a liquid, the calculation unit 140 may calculate the volume of the liquid by using the tableware information including a location where the liquid and the corresponding space 510 contact each other, and a size, depth, and capacity of the space 510.
When a liquid food such as a ‘soup’ in a specific space 510 such as ‘A’ area shown in
In some embodiments, additional correction may be performed on a single tableware 500 with no space division as shown in
And the calculation unit 140 may calculate the volume of the whole food using the extracted whole food image data, and correct the volume of each food by comparing the total volume of each food calculated in the step of calculating volume and the volume calculated through the whole food image data.
In some embodiments, the step of calculating volume (step S50) may further include a step of calculating volume of stacked food (step S51).
More specifically, a plurality of foods may be stacked in a specific space 510 within the tableware 500. Therefore, when it is recognized that different foods are stacked in a specific space 510 within the tableware 500, the calculation unit 140 calculates the volume of the foods located on the lower side by using image data of the foods located on the upper side, the calculated volume information, and the size, depth, and capacity information of the space 510.
Referring to
To this end, the height of the area where the food (e.g., boiled rice) located on the lower side is covered by the food (e.g., a fried egg) located on the upper side may be set as an average of the height of the area where the food (e.g., boiled rice) located on the lower side is not covered by the food (e.g., a fried egg) located on the upper side, but it is not limited thereto.
In some embodiments, data on foods (e.g., fried eggs, baked fish, seasoned laver, etc.) likely to be stacked on top of other foods may be previously stored in the database 170 of the computer. For example, data on foods mainly staked on the upper side of the food may include volume data or weight data, and through this, the calculation unit 140 may quickly calculate the volume of the food stacked on the upper side.
Meanwhile, the step of calculating volume (step S50) may further include a step of correcting, by the correction unit 150, a noise component photographed together with a tableware in a tableware image (step S52). That is, it may further include a step that the correction unit 150 detects a noise component that is not food and has a volume in the image data of the tableware containing food and performs correction to exclude the volume of the detected noise component from the calculated food volume in the process of calculating volume of each food.
Here, the noise component is not food and has a volume, and the volume of the food may be calculated larger than the actual volume due to the volume of the noise component. Therefore, the correction unit 150 may recognize the noise component in the photographed image in order to accurately calculate the volume of the food and perform correction to exclude the volume of the noise component from the volume of the food calculated by the calculation unit 140.
For example, a hand or cutlery (e.g., a spoon, a fork, a knife, etc.) may be a noise component that is not food and has a volume, and a lid of a dairy product may correspond to this, but it is not limited thereto. For example, noise information on an object corresponding to a noise component may be previously stored in the database 170 of the computer, and the correction unit 150 may recognize a noise component in the photographed image and perform correction to exclude the volume of the noise component from the volume of food calculated by the calculation unit 140.
In one embodiment, the correction unit 150 may pre-learn the image data for empty tableware 500, hands, or cutleries, and if tableware image includes hands or cutleries, the correction unit may recognize them and perform correction to exclude the volume of the hands or cutleries.
Next, referring to
The database 170 of the computer has stored food information including weight per volume for each food, calorie information per weight for each food, and nutrition facts per weight for each food.
Foods have different weight per volume depending on the type, and their calories are different from each other. Accordingly, in an embodiment of the present invention, food information including weight per volume for each food is stored in the database 170, and using this, the calculation unit 140 calculates weight of food from volume information of food.
Using this, after the step of calculating weight (step S60), the calculation unit 140 may calculate calories of food accommodated in a user's serving tableware using the calculated weight information of each food, and in addition, calculate nutritional component information eaten by the user, and calculate additional specific information such as allergy precautions based on food or nutritional components included in the food.
Besides, the calculation unit 140 may calculate price information for corresponding food by using the type of food and the volume or weight of each food.
The calculation unit 140 may provide information included in the above-described meal information to a display unit 210 of the device for measuring food 200 or a display unit 310 of the user terminal 300. The UI of the screen provided on the display units 210 and 310 will be described later.
For example, the calculation unit 140 may provide different information to be displayed on the display unit 210/310 of the food serving section and the display unit 210/310 of the tableware return section. For example, the display unit 210/310 of the food serving section may display information on the type of food, information on calories, nutritional component information, allergy information and price information, and the display unit 210/310 of the tableware return section may additionally display information on an amount actually eaten in addition to the above information. However, the meal information calculated by the calculation unit 140 is not limited thereto, and various information may be displayed without limitation by using photographed information or information stored in the database 170.
Accordingly, the taker may make payment through a payment unit 240 of the device for measuring food 200 or the user terminal 300 by cash, card, account transfer, QR code tagging, RFID tagging, or facial recognition, etc. according to the price information provided.
In an embodiment, a restaurant may have a weighing device (not shown) capable of measuring the weight of the tableware 500, but it is not limited thereto, and it may further include a step that the receiving unit 110 receives the weight information of the tableware 500 measured from the weighing device (not shown) provided in the restaurant, and a step that the correction unit 150 corrects the weight of each food by matching the total weight of each food and the weight of the empty tableware calculated with the received weight information.
The above-described correction step can be selectively adopted and used, and since the weight may be measured differently from the actual weight due to the user's hand or various factors in the process of measuring the weight, if the weight information is measured differently to the extent that it exceeds threshold value, the correction step may not be performed.
Hereinafter, a method for measuring food according to another embodiment of the present invention that uses the user terminal 300 as the device for measuring food 200 will be described with reference to
The method for measuring food according to another embodiment of the present invention is performed by a computer, wherein the computer refers to a server for measuring food 100 or a user terminal 300. That is, the photographing may be performed by the user terminal 300, but other steps may be performed by the server for measuring food 100 or the user terminal 300.
Here, the user terminal 300 is a portable mobile terminal, in which a service application related to a method for measuring food is installed. The user terminal 300 may acquire image data and depth data. To this end, the user terminal 300 may include at least one of an RGB camera, a 2D camera, a 3D camera, a Time of Flight (ToF) camera, a light field camera, a stereo camera, an event camera, and an infrared camera. For example, the user terminal 300 may include all user devices capable of installing and executing a food measurement application related to the server for measuring food 100 or the device for measuring food 200.
First, referring to
Here, food may be photographed according to a request to shoot a photograph of a service application installed in the user terminal 300, and a guide for a method of photographing food, an angle, etc. may be provided to the service application user as by video or sound.
In an embodiment, the service application may request to shoot food two times or more from different angles when not using the 3D camera or ToF camera.
Subsequently, referring to
In an embodiment, in the step of receiving the tableware image (step S10), tilt information of the user terminal 300 measured through a gyro sensor (not shown) of the user terminal 300 may be received when food is photographed through the photographing unit 350 of the user terminal 300. The correction unit 150 may correct the tilt of the photographed image data of tableware containing food by using the tilt information of the user terminal 300.
According to an embodiment of the present invention, since a certain angle of tilt may occur in the process of photographing food through the photographing unit 350 by the user holding the user terminal 300, the above-described steps may be performed to correct the image data of tableware containing food into a horizontal state.
Next, referring to
Since the tableware image photographed by the user terminal 300 may not contain only food, only food image data is extracted from the image at this step for accurate volume calculation.
The extraction unit 120 accurately extracts food from the background using a function of recognizing and extracting food from the background, and separates and extracts each image data of two or more foods when two or more foods are included, and in this process, a model learned using an artificial intelligence algorithm may be used.
Next, referring to
The determination unit 130 determines the food type of each food image data extracted by the extraction unit 120 using a model learned using an artificial intelligence algorithm. In some embodiments, if a plurality of determination results with similar accuracy are obtained, the determination unit 130 may output the plurality of determination results to the user terminal 300 and request the user to input or select a suitable result through the service application of the user terminal 300.
Next, referring to
The step of calculating volume (step S50) may include a step of correcting volume (step S53). In an embodiment, a background target may be used for volume correction in the step of correcting volume (step S53).
In the step of correcting volume (step S53), the determination unit 130 may recognize the background target included in the received tableware image, and the correction unit 150 may correct the volume of each food calculated in the step S50 based on size information and location information of the recognized background target.
The correction unit 150 may perform volume correction of each food by comparing the background target and height information of each food, or may perform volume correction of each food by comparing the background target and size information of each food.
Here, the background target is photographed with food or separately photographed through the photographing unit 350, and a tableware 500 containing food; a table; a cutlery; and a user's fingers, etc. may be used as the background target. However, it is not limited thereto, and the background target may be applied to any target having a usual size. For example, the background target may be applied to any target whose size is determined, such as coins, bills, and smartphones (by product type). In some embodiments, information on the size of a background target photographed through the photographing unit 350 may be previously stored in a database 170.
As a first example, the service application may request the user to photograph the user's hand at a certain distance through the photographing unit 350. Alternatively, the service application may request to measure and input the actual size of the user's hand/finger.
In addition, the receiving unit 110 analyzes the user's hand image photographed and stores information on the actual size of the user's hand in the database 170.
Thereafter, when it is recognized that the user's hand is included in the received tableware image, the determination unit 130 may set the user's hand as a background target, and the correction unit 150 may correct the volume of each food calculated by the calculation unit 140 based on the location information and size information of the background target.
As a second example, the service application may request the user to photograph tableware 500, tables, cutleries, etc. used at home through the photographing unit 350 at a predetermined distance. In this case, the table may refer to a table or a rice table where the user places the tableware 500 and has a meal.
Specifically, the computer may analyze the image of the tableware 500 used by the user at home, photographed through the photographing unit 350 of the user terminal 300, and store information on the tableware 500 including size, depth, and capacity information of the corresponding tableware 500 in the database 170.
More specifically, since height information for each pixel is stored in the image photographed through the photographing unit 350 of the user terminal 300 according to an embodiment of the present invention, when the image data for the empty tableware 500 is received, the calculation unit 140 may calculate size, depth, and capacity information for the tableware using height information for each pixel of the image data of the tableware, which means to store them as the tableware information in the database 170.
In addition, as a result of analyzing food image, if food contained in the tableware 500 exists among the food image, the calculation unit 140 performs correction of the calculated volume of each food by using information on the size, depth, and capacity of the tableware 500 in which the corresponding food is accommodated.
As described above, if the size, depth, and capacity information of the corresponding tableware 500 is known, a more accurate result may be obtained by performing a correction step.
In some embodiments, the service application may request the user to measure the actual size of the tableware 500, the table, and the cutlery, etc. (e.g., information on the width, height, or perimeter of the table) and input them into the user terminal 300.
As a third example, a case of using a background target that is not related to food will be described. For example, the user may photograph a 500 won coin along with food, and input through the service application that a 500 won coin has been photographed as a background target.
Accordingly, the determination unit 130 recognizes the 500 won coin (background target) included in the tableware image including the received food, and the correction unit 150 corrects calculated volume of each food based on size information of the 500 won coin (background target).
In addition, the accuracy may be increased by correcting the volume by variously utilizing location information (restaurant, food service, etc.), weather information (ultraviolet rays, reduction of food type), user's status information (during diet, exercise, fighting disease, medication, etc.), personal preference information, and peripheral device information.
As a fourth example, the computer may build up big data by storing image data photographed through the user terminals 300 of a plurality of users, information calculated through the image data, and information on restaurants that match the places where the images were photographed.
For example, if user D photographed food at a location matching restaurant B and transferred the food image data, the location information is matched with big data to determine that user D was located at restaurant B, and correction for the calculated volume information may be performed by inquiring the previous result data matching the ordered menu.
In this regard, the opposite is also possible. More specifically, if data about food is stored first and it is known what kind of food a user ordered and ate when the user is paying, inversely, the stored data may be retrieved so as to record volume and nutrition information.
Next, referring to
Using the method for measuring food according to according to an embodiment of the present invention, the computer may measure the amount of food served (meaning the amount of food before meals) and the amount of leftover for each taker at a restaurant or at home, and using this, may calculate the amount of food eaten.
In addition, the computer may generate operating information for restaurants and management information for takers by using the amount of food served, the amount of leftover, and the amount of food eaten for takers eating at restaurants or at home, and the management information for takers may include, for example, user's nutritional status, eating habits, and whether the user eats only what one wants, etc.
Hereinafter, a method of providing restaurant operation information and management information of a taker according to an embodiment of the present invention will be described with reference to
First, referring to
Next, referring to
Next, referring to
Through steps S110 to S120, the computer may obtain information such as how much the taker ate meal and how much the taker left meal by calculating the amount of food served, the amount of food eaten, and the amount of leftover of the taker for each food menu provided by the restaurant.
In addition, the computer according to an embodiment of the present invention may perform the following operations to improve the accuracy of calculating the amount of food.
In an embodiment, the calculation unit 140 verifies whether the sum of the amount of food eaten and the amount of leftover of the taker calculated through the above steps matches the amount of food served, and if there is a discrepancy according to the verification result, the calculation unit 140 may correct at least one of the amount of food served, the amount of leftover, and the amount of food eaten calculated through the above steps so that the sum of the amount of food eaten and the amount of leftover matches the amount of food served, and records the result in the database 170.
In an embodiment, the calculation unit 140 verifies whether the sum of the total amount of food eaten and the total amount of leftover of takers at a restaurant calculated through the above steps matches the total amount of food served, and if there is a discrepancy according to the verification result, the calculation unit 140 may correct at least one of the total amount of food served, the total amount of leftovers, and the total amount of food eaten calculated through the above steps so that the sum of the total amount of food eaten and the total amount of leftover matches the total amount of food served, and records the result in the database 170.
In addition, the calculation unit 140 verifies whether the sum of the total amount of food eaten and the total amount of leftovers of takers at the restaurant calculated through the above steps matches the total amount of food served, and records the result in the database 170. More specifically, the calculation unit 140 verifies whether the difference value between the total amount of food served and the total amount of leftovers of the takers eating at the restaurant matches the total amount of food eaten.
In addition, the calculation unit 140 performs verification by comparing the total amount of food prepared at the restaurant with the sum of the total amount of leftovers generated by the takers at the restaurant, the total amount of remaining food not served at the restaurant, and the total amount of food eaten by the takers at the restaurant, and if there is a difference according to the verification result, the calculation unit 140 may correct at least one of the total amount of leftovers, the total amount of remaining food, and the total amount of food eaten so that the total amount of food prepared at the restaurant matches the sum of the total amount of leftovers generated by the takers at the restaurant, the total amount of remaining food not served at the restaurant, and the total amount of food eaten by the takers at the restaurant, and record the result in the database 170.
Here, the amount of leftover may be the amount of food left by the taker after being served to the taker, and the amount of remaining food may be the amount of food left without being served to the taker, and the amount of remaining food may be calculated through the above steps or through a separate method.
In addition, the database 170 may store a result of measuring the total weight of the amount of food prepared for each serving at the restaurant, the total volume (weight) of the remaining food, and the total weight of the leftovers, and the calculation unit 140 performs verification by comparing the total volume (weight) of the prepared food with the sum of the total volume (weight) of the amount of leftovers, the total volume (weight) of the amount of remaining food, and the total amount of food eaten by the takers at the restaurant calculated through the above steps, and if there is a difference according to the verification result, the calculation unit 140 may correct the total amount of food eaten by the takers calculated through the above steps so that the total amount of food prepared at the restaurant matches the sum of the total amount of leftovers generated by the takers at the restaurant, the total amount of remaining food not served at the restaurant, and the total amount of food eaten by the takers at the restaurant, and record the result in the database 170.
In this way, the calculation unit 140 may identify and improve problems by performing verification using various types of information and recording the accuracy due to calculation/correction.
Meanwhile, in order to accurately calculate the amount of food eaten, pictures must be taken before and after meal, but there may be a case where the user does not accidentally take a picture after meal.
Therefore, if the calculation unit 140 receives image data of tableware containing food before a meal and then does not receive a photographed post-meal tableware image data until a preset time lapses, the calculation unit determines that the post-meal tableware image data has not been photographed, and calculates the user's expected amount of food eaten.
For example, the calculation unit 140 may calculate the expected amount of food eaten by the user by regarding that the user has not left a leftover, or may calculate the expected amount of food eaten by the user based on information such as an average amount of food eaten by the user, status information (e.g., during diet, increasing weight, exercise), degree of hunger, user preference about menus included in the pre-meal food image, and taste information of menus stored in the database 170, and store the calculated result in the database 170.
Next, referring to
Next, referring to
Step S140 means generating operation information that helps the restaurant operation.
More specifically, a serving amount control module 181 of the management unit 180 uses information on the total prepared amount for serving prepared for serving at the restaurant, the total amount of food served for the takers who received serving, and the number of remaining takers who did not receive serving to calculate target serving amount per person for remaining takers and provide it to a restaurant manager device (not shown).
In this case, the restaurant manager may refer to a person in charge who designs a menu in the restaurant and serves food directly to the takers, a manager who manages the restaurant, or a dietitian of the restaurant. A restaurant manager device such as a display device 30 may be installed around food prepared to be served in front of the restaurant manager of the restaurant.
Accordingly, when the serving amount control module 181 provides information on the target serving amount per person for remaining takers on the restaurant manager display, the restaurant manager may use this to control the serving amount for the remaining takers. For example, the restaurant manager may raise the target serving amount per person for the remaining takers if the prepared amount for serving is sufficient compared to the number of remaining takers, and reduce the target serving amount per person for the remaining takers if the prepared amount for serving is insufficient compared to the number of remaining takers.
In an embodiment, the operation information generation step (step S140) may include a step that a serving group management module 183 checks the affiliation of each personnel to be served, stores the amount of food served, the amount of leftovers, and the amount of food eaten for each affiliation and using this, generates information for each affiliation including at least one of preferred food, non-preferred food, amount of leftovers, nutritional status for each affiliation; and a step that extracts correlation information by matching the weather, the day of the week, schedule information for each affiliation, and the information for each affiliation.
Accordingly, according to an embodiment of the present invention, it may be not only possible to identify taste preference for food for each affiliation using information on preferred food and non-preferred food depending on the affiliation of the takers, but also to prepare an accurate guide for the next serving menu by analyzing correlation between the weather, the day of the week, schedule information for each affiliation, and an amount of food eaten.
In an embodiment, in the operation information generation step (step S140), if the serving amount is similar for each restaurant, but the difference in the amount of leftovers differs by more than a certain amount, the management unit may determine that the restaurant having smaller amount of leftovers has excellent food ingredient grades and cooking methods, and provide information on food ingredient supply and demand and cooking methods of the corresponding restaurant to restaurant manager devices of other restaurants.
Accordingly, the management unit may exert an effect of improving quality of restaurants by providing information on food ingredient supply and demand and cooking methods of the restaurant having smaller amount of leftover.
In an embodiment, the management information generation step (step S150) includes a step that a taker management module 185 generates information for each taker including at least one of preferred food, non-preferred food, an average amount of leftover, nutritional status, an average amount of food eaten by using body information, an amount of food served, amount of leftover, and an amount of food eaten for each taker.
In addition, the management information generation step also includes a step that the taker management module 185 calculates physical strength management information and risk of obesity for each taker using the information for each taker, and if the amount of food eaten of a taker not set as a diet state decreases below a certain amount for a certain period of time or the amount of leftover of the taker increases above a certain amount for a certain period of time, the taker is set as a subject of attention.
Therefore, if the amount of food eaten of a taker not set as a diet state decreases below a certain amount for a certain period of time or the amount of leftover of the taker increases above a certain amount for a certain period of time, the taker management module 185 determines that the taker has a problem causing a decrease in appetite, and set the taker as a subject of attention. Through this, there is an effect that it is possible to prepare countermeasures such as consultation by a manager by early detection of danger signs before an accident occurs to personnel such as protective-concerned soldiers in a place such as the military.
In an embodiment, the operation information generation step (S140 step) includes a step that average preferred food, non-preferred food, and an amount of food eaten for takers are calculated and using this, a serving menu recommendation module 187 derives one or more recommended menus and prepared amount for serving of the restaurant and provides them to the restaurant manager device.
In addition, in order to implement such an algorithm, a degree of relevance for each food is stored in the database 170 of the computer. For example, if foods A, B, and C are highly related and the takers of the corresponding restaurant prefer food B, it may be determined that foods A and C are likely to be preferred.
At this time, the serving menu recommendation module 187 may derive a recommended menu using preference information and taste evaluation score of food menu to be described below.
In an embodiment, a serving menu evaluation module 189 of the management unit 180 generates operation information and management information based on the amount of food served, the amount of leftover, and the amount of food eaten, and evaluates a preference of each food menu and a taste evaluation score of the food served itself by using an amount of change in the amount of food served and an amount of change in the amount of leftover of each of the foods served. With this information, the restaurant manager can check whether there is a problem with the food preparation process.
The above-described configuration evaluates a score by using these features, and more specifically, it is determined based on the following criteria.
First, when a serving amount of a particular food is greater than a reference value and an amount of leftover is less than a reference value, the serving menu evaluation module 189 increases a preference of the corresponding food menu and increases a taste evaluation score of the corresponding food served itself.
Second, when the serving amount of a particular food is greater than the reference value and the amount of leftover is greater than the reference value, the serving menu evaluation module 189 increases the preference of the corresponding food menu and decreases the taste evaluation score of the corresponding food served itself.
Third, when the serving amount of a particular food is less than the reference value and the amount of leftover is less than the reference value, the serving menu evaluation module 189 decreases the preference of the corresponding food menu and increases the taste evaluation score of the corresponding food served itself.
Fourth, when the serving amount of a particular food is less than the reference value and the amount of leftover is greater than the reference value, the serving menu evaluation module 189 decreases the preference of the corresponding food menu and decreases the taste evaluation score of the corresponding food served itself.
In an embodiment, the taker management module 185 of the management unit 180 generates nutrition facts eaten by each taker based on the amount of food eaten by the takers, and derives and provides nutritional intake status information based on the nutrition facts generated at regular time intervals. In this case, the taker management module 185 may provide the derived information to a terminal of the taker, that is, the user terminal 300, a terminal of a guardian of the taker, a terminal of a manager of the taker, and a terminal of the restaurant manager, etc. In addition, rewards or penalties are provided according to changes in nutritional status and amount of leftover of each taker derived using the above information.
In addition, when the taker is a student or a patient who needs management by a guardian, the taker management module 185 of the management unit 180 may provide a pre-meal tableware image and a post-meal tableware image of the taker obtained through the photographing unit 250 to the terminal of the guardian of the taker. The taker management module 185 determines the nutritional status and eating habits of the student or patient based on their serving menu, data of an amount of food eaten, and an amount of basic metabolism.
The taker management module 185 may provide the information obtained and determined in this way to the user terminal 300 and the terminal of guardian of the taker, and at the same time, based on the nutritional status and eating habits of each taker, may derive a recommended menu good for the corresponding taker to eat at home and may recommend a nutritional supplement according to the situation.
In addition, the user's body information may be input through a service application installed in the user terminal 300, and the user's basic metabolism may be calculated using the body information.
In addition, by using the user's nutritional status and eating habit information derived as above, one or more food menus matching the user may be derived and provided through a service application, or the information may be provided in conjunction with other food ordering applications.
In this case, the service application may directly receive input of information on the user's preferred food and non-preferred food from the user, and may receive input of information on the user's allergy, disease, and the like and exclude foods that the user should not eat from the recommended menu.
In addition, using such information, the determination unit 130 may provide a warning (notification) to the service application when food that the user should not eat is included in the food image photographed through a photographing unit 55 of a portable device 50, thereby preventing users from accidentally eating the food.
The computer may collect data on the amount of food before meal, the amount of food eaten, and the amount of leftover at each meal of the user through the user terminal 300 and store the data in the database 170, thereby constructing information such as the user's nutritional status and eating habits, and may collecting data on the user's body information, and user's activity index, activity information, and exercise information collected through various health and exercise applications to construct the user's health care information.
In addition, by interlocking with other health management applications and medical information applications to collect user's medical information, the user's nutritional status, eating habits, health care information, and medical information may be used to derive and provide customized diet and health supplement information for the user, and products related thereto may be directly sold through a service application.
Hereinafter, a device and a server for implementing a method for measuring food according to an embodiment of the present invention will be described.
A server for measuring food 100 according to an embodiment of the present invention will be described with reference to
Referring to
However, in some embodiments, the server 100 may include fewer or more components than the components illustrated in
The receiving unit 110 receives a tableware image photographed through one or more photographing units 250.
The extraction unit 120 extracts food image data accommodated in each space 510 of the tableware 500 from the image data of tableware containing food.
The determination unit 130 determines a type of each food based on the extracted food image data.
The calculation unit 140 calculates a volume of each food by using height information for each pixel of the extracted food image data.
The database 170 stores an empty tableware image, tableware information including size, depth, and capacity of a plurality of spaces 510 formed in the tableware 500, and food information including weight per volume for each food.
Referring to
The communication unit 190 performs communication so as to receive a photographed tableware image from the photographing unit 250 installed in the restaurant, and may play a role to transmit various information calculated and generated by the server 100 to a person in charge or the terminal of the user.
The authentication unit 160 identifies a user by authenticating user information received from devices such as the device for measuring food 200 and a kiosk device 30, and loads various information on the user.
In addition, information on the weight, calories, etc. of foods in the serving tableware calculated through the calculation unit 140 of the server 100 is provided to the terminal of the user authenticated through the authentication unit 160. In this case, the information may be provided through a service application provided by the server 100 itself.
Since the server for measuring food 100 according to an embodiment of the present invention described above differs from the method for measuring food described with reference to
Hereinafter,
The device for measuring food 200 may be provided at a place where food is served, and includes a receiving unit 110, an extraction unit 120, a determination unit 130, a calculation unit 140, a correction unit 150, an authentication unit 160, a database 170, a management unit 180. a communication unit 190, a display unit 210, a payment unit 240, and a photographing unit 250. However, in some embodiments, the device for measuring food 200 may include fewer or more components than the components illustrated in
Referring to
In addition, the measuring area 220 or the plate 230 include a weight measuring device (not shown) formed therein to measure the weight of the serving tableware placed on the upper surface thereof.
The receiving unit 110 receives a tableware image photographed through one or more photographing units 250. Here, the photographing unit 250 is positioned so that an upper surface of the tableware 500 may be photographed from the upper side over the tableware 500, and if an identification information sensing unit 251 is present, the identification information sensing unit 251 may be positioned so that identification information of the tableware 500 may be detected from the upper side, the lower side, or the side at the same time as the tableware 500 is photographed by the photographing unit 250.
The food measuring device 200 may include a display unit 210, receive input of user authentication information for authenticating a user, and display a photographed image through the display.
In addition, when the device for measuring food 200 is provided in the form of
In some embodiments, referring to
In some embodiments, referring to
In the process of moving the tableware 500 by the conveyor belt 270, the photographing unit 250 may photograph the tableware 500 from the upper side over the tableware 500.
In some embodiments, the device for measuring food 200 may include a photographing unit 250, an identification information sensing unit 251, and an authentication unit 160, and the photographing unit 250 may be located in a relatively highest region, the authentication unit 160 may be located between the photographing unit 250 and the identification information sensing unit 251, and the identification information sensing unit 251 may be located in a relatively lowest region. Since the device for measuring food 200 has such a structure, the photographing unit 250, the identification information sensing unit 251, and the authentication unit 160 may obtain necessary information from the taker and the tableware 500 just by the taker stopping in front of the device for measuring food 200. And this may reduce inconvenience of the taker.
The photographing unit 250 is positioned in the highest region for the purpose of photographing the tableware 500 from above. Assuming that the authentication unit 160 recognizes the face of the taker, when the taker holds the tableware 500, the tableware 500 is located below the face of the taker, so the authentication unit 160 may be positioned relatively above the identification information sensing unit 251. Here, identification information 530 of the tableware 500 may be located on the side of the tableware 500 and may be recognized by the identification information sensing unit 251 when the taker stands while holding the tableware 500.
Meanwhile, the identification information sensing unit 251 and the authentication unit 160 may not always operate, but may only operate when the tableware 500 is photographed by the photographing unit 250, and through this, the recognition of the tableware 500 by the photographing unit 250 becomes a kind of enabler, thereby effectively operating the identification information sensing unit 251 and the authentication unit 160.
Referring to
The device for measuring food 200 may be tilted with its upper part toward the front, and the photographing unit 250a for photographing food or tableware containing food may be positioned in the tilted area of the device for measuring food 200 to photograph the tableware 500 located below. Although the photographing unit 250b for photographing biometric information of a taker is shown not to be in the tilted area, it may be located in the tilted area.
And when the taker photographs the tableware 500 containing food with the device for measuring food 200 or the user terminal 300, the screen of
For example, a tableware image containing food photographed by the photographing unit 250/350 may be displayed, and information on a type of food and calories according to the type of food may be displayed. And when a beverage and the like is contained in a cup, the height of the beverage in the cup may be displayed. In addition, information on exercise and exercise time required to burn calories from eating food may also be displayed together. Besides, nutrition facts and price information of the food to be consumed may also be displayed together. In addition, since
Since the detailed contents of the device for measuring food 200 configurations according to an embodiment of the present invention described above differs from the method for measuring food described with reference to
The method for measuring food according to an embodiment of the present invention described above can be embodied as a program (or application) to be executed in combination with a server which is hardware, and can be stored in a medium.
The above-described program may include a code coded in a computer language such as C, C++, JAVA, and machine language readable by a computer's processor (CPU) in order for the computer to read the program and execute the methods embodied as a program. Such code may include a functional code related to a function and the like that defines necessary functions for executing the methods, and may include a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such code may further include a code related to a memory reference to which location (address number) of the internal or external memory of the computer should be referenced for additional information or media required for the processor of the computer to execute the functions. In addition, when the processor of the computer needs to communicate with any other computer or server in a remote location in order to execute the functions, the code may further include a communication related code for how to communicate with any other computer or server in a remote location using a communication module of the computer and what information or media to be transmitted and received during communication and the like.
The storage medium refers not to a medium that stores data for a short moment, such as a register, cache, memory, etc., but refers to a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on various servers to which the computer can access, or in various recording media on the user's computer. In addition, the medium may be distributed over computer systems connected through a network, and may store a code readable by the computer in a distributed manner.
In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but it will be understood by those of ordinary skill in the art that the present invention may be implemented in other specific forms without changing the technical spirit or essential features. Therefore, the embodiments described above are exemplary in all aspects and should be understood as non-limiting.
