Patent Grant
11928769
This disclosure relates generally to appliances, and more particularly to hyperspectral and multispectral imaging within kitchen appliances.
Smart kitchen appliances increasingly rely on computer vision (CV) to recognize foods to enhance user experiences. However, CV algorithms are computationally intensive, requiring expensive hardware (e.g., graphics processing units) to be integrated into the kitchen appliance. Further, foods represent a diverse category of potential images and misidentification of food is a common issue. Further, even when CV systems correctly identify foods, the algorithm may not assess food quality in the same way a human user would. It may be thus useful to provide techniques to improve kitchen appliance devices.
In particular embodiments, the one or more processor(s) 104 may be operably coupled with the memory 106 to perform various algorithms for a kitchen appliance. Such programs or instructions executed by the processor(s) 104 may be stored in any suitable article of manufacture that includes one or more tangible, computer-readable media at least collectively storing the instructions or routines, such as the memory 106. The memory 106 may include any suitable articles of manufacture for storing data and executable instructions, such as random-access memory (RAM), read-only memory (ROM), rewritable flash memory, hard drives, and so forth. Also, programs (e.g., an operating system) encoded on such a computer program product may also include instructions that may be executed by the processor(s) 104 to enable the kitchen appliance 102 to provide various functionalities.
In particular embodiments, the sensors 108 may include, for example, one or more cameras (e.g., depth cameras), touch sensors, microphones, motion detection sensors, thermal detection sensors, light detection sensors, time of flight (ToF) sensors, ultrasonic sensors, infrared sensors, or other similar sensors that may be utilized to detect the physical state of the load (e.g., its temperature profile) or various user inputs (e.g., user voice inputs, user gesture inputs, user touch inputs, user instrument inputs, user motion inputs, and so forth). RF and microwave electronics 110 may include, for example, any number of magnetrons, power amplifiers, signal generators, waveguides, antennas, integrated circuits, transistors, solid-state amplifiers, and/or one or more other devices that may be collectively suitable for directing electromagnetic energy (e.g., microwave energy) to a load (e.g., food, solids, liquids, and so forth) that may be disposed inside the interior of the kitchen appliance 102. The display panel 112 may include any display architecture (e.g., AMLCD, AMOLED, micro-LED, and so forth), which may provide further means by which users may interact and engage with the kitchen appliance 102.
In particular embodiments, the input structures 114 may include any physical structures utilized to control one or more global functions of the kitchen appliance 102 (e.g., pressing a button to power “ON” or power “OFF” the kitchen appliance 102). The network interface 116 may include, for example, any number of network interfaces suitable for allowing the kitchen appliance 102 to access and receive data over one or more cloud-based networks (e.g., a cloud-based service that may service hundreds or thousands of the kitchen appliance 102 and the associated users corresponding thereto) and/or distributed networks. The power source 118 may include any suitable source of power, such as a rechargeable lithium polymer (Li-poly) battery and/or an alternating current (AC) power converter that may be utilized to power and/or charge the kitchen appliance 102 for operation. Similarly, the I/O interface 120 may be provided to allow the kitchen appliance 102 to interface with various other electronic or computing devices, such as one or more auxiliary electronic devices.
Certain technical challenges exist for imaging objects within kitchen appliance 102 in a way that allows for accurate identification of the object and/or an accurate quality assessment of the object. In general, illumination within consumer appliances are tailored around making objects inside the interior of the appliance look aesthetically pleasing to the user. For example, within a refrigerator, the illumination source may be tailored to create an aesthetic appearance of food to a user through the use of lights that maximize the color rendering index. While computer vision (CV) within kitchen appliance 102 may be employed to identify the food within the interior, due to a wide variety of possible foods, the CV system may incorrectly identify the food. Further, CV systems may be trained to assess the quality of each food within the interior of kitchen appliance 102. However, the illumination source within kitchen appliance 102 may not be suitable for the CV system to accurately conduct a quality assessment of the food. The solution presented by the embodiments disclosed herein to address this challenge may be hyperspectral imaging (HSI) and/or multispectral imaging (MSI) techniques for imaging a scene within an appliance (e.g., food within a refrigerator). Another technical challenge may be that the light source and sensors within kitchen appliance 102 are not suitable for performing HSI and/or MSI. The solution presented by the embodiments disclosed herein to address this challenge may be controlled illumination techniques within kitchen appliance 102 to permit HSI and/or MSI imaging of a scene.
Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of the embodiments may include a hyperspectral and/or multispectral imaging system of kitchen appliance 102. Another technical advantage of the embodiments may include automatic food recognition through hyperspectral imaging (HSI) and/or multispectral imaging (MSI). Another technical advantage of the embodiments may include automated cooking through HSI-enabled and/or MSI-enabled monitoring of food doneness. A further technical advantage of the embodiments may include food quality monitoring through hyperspectral and/or multispectral imaging. Another technical advantage of the embodiments may include food cataloging and/or tracking by a kitchen appliance using HSI and/or MSI techniques. Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages. One or more other technical advantages may be readily apparent to one skilled in the art in view of the figures, descriptions, and claims of the present disclosure.
As used herein, “hyperspectral imaging (HSI)” may refer to a technique for analyzing images at plurality of spectral bands, wherein the resulting images may display a large number of contiguous spectral bands. As used herein, “multispectral imaging (MSI)” may refer to a technique of capturing image data within specific wavelength ranges, wherein the resulting images provide discrete and discontinuous portions of the spectral range. It is understood that the range of illumination wavelengths may not be restricted to visible wavelengths and may include UV and infrared radiation (IR) wavelengths.
In particular embodiments, kitchen appliance 102 may illuminate a scene with narrowband light at a specific wavelength. As used herein, “narrowband light” may refer to light at a specific wavelength of optical radiation. As an example and not by way of limitation, a sensor, such as a broadband camera, may capture an image before the wavelength is changed. In this example, sequential illumination with multiple narrowband light-emitting diodes (LEDs) may be captured with a monochrome camera, wherein the number of LEDs and the wavelengths emitted by the LEDs may be optimized for the particular application. As an example and not by way of limitation, a bank of LEDs emitting at a plurality of different discrete wavelengths may sequentially illuminate the interior of the kitchen appliance 102 when the appliance door is closed, and a broadband monochromatic camera may record images and reconstruct a hyperspectral data cube. As used herein, “data cube,” may refer to a series of two-dimensional (2D) images taken at different wavelengths. In particular embodiments, each 2D image in a hyperspectral data cube may reveal the scenes intensity at a given wavelength. Within the 2D image, the reflectance Rij at the ijth pixel may be calculated as:
In Equation 1, Iij is is the intensity of the ijth pixel as captured by the camera in the 2D image and Lij(λ) is the scene illumination at that pixel's location at wavelength λ. It is important to control or at least know Lij at all wavelengths in order to extract meaningful reflectance data, which is possible by properly specifying the light source within the appliance.
In particular embodiments, one method for hyperspectral imaging (HSI) may be to illuminate the scene with a broadband light and then constrain the field of view to an approximately one-dimensional (1D) line across the scene. As an example and not by way of limitation, a dispersive or diffractive optic may be used to spread the line-image into corresponding spectral components, wherein different parts of the line-image correspond to different spectra according to the composition of the object being imaged. In this example, the result may be a 2D image captured by one or more sensors 108, wherein the Y-axis may represent image intensity variation in the Y dimension, and the X-axis may indicate intensity variation of each wavelength. As an example and not by way of limitation, the line-image may be scanned to select one or more different lines within the image, and the data from successive scans by the one or more sensors 108 may be combined into a three-dimensional (3D) hyperspectral data cube, as demonstrated in
In particular embodiments, illumination source 210 may illuminate one or more objects 220, wherein mirror 230 may scan a field of view. As an example and not by way of limitation, light may pass through one or more objective lenses 240 to focus the light onto slit 250, wherein slit 250 (e.g., 20 μm slit) may select a line from the reflected image. As an example and not by way of limitation, slit 250 may be used in place of or in addition to one or more cylindrical lens arrays or other suitable lens array. As another example and not by way of limitation, slit 250 may be moveable and/or adjustable to capture particular positions of objects 220. Further, slit 250 may be reflective to act as a mirror, or alternatively have absorbent properties to reduce background radiation and/or other background interference.
In particular embodiments, mirror 230 may be a rotating mirror, wherein a separate image may be reflected for each angle of rotation of mirror 230. Light reflected by mirror 230 may then pass through one or more collimating lens(es) 260 and be diffracted by diffraction grating 270, wherein the line from the reflected image may be spread into each component wavelength for capture by processor 104 of one or more sensors 108 of kitchen appliance 102. As an example and not by way of limitation, the one or more sensor(s) 108 may include a monochrome camera with high sensitivity. In particular embodiments, multiple images may be generated by rotating mirror 230, wherein each image captured by processor 104 of the one or more sensors 108 of kitchen appliance 102 may be stored in memory 106 of kitchen appliance 102, and the compilation of stored images may be used to generate a three-dimensional (3D) data cube 290.
In particular embodiments, spectral and/or spatial information may be extracted by slicing the 3D data cube 290 in one or more directions. As an example and not by way of limitation, 3D data cube 290 may be collected in approximately three seconds and is typically 640×480×600 pixels (e.g., a file size of approximately 100 MB). In particular embodiments, 3D data cube 290 may be formed by combining a set of monochromatic or nearly monochromatic images taken at a set of wavelengths that provide useful spectral information about object 220. In this example, each image in 3D data cube 290 may be acquired by using a different distinct wavelength. Further, in this example the imaging sensor (e.g., camera) must have sensitivity corresponding to the wavelengths emitted by the LED source. In particular embodiments, illumination source 210 may be used in combination with a series of monochromatic or nearly monochromatic filters. In this example, each image in 3D data cube 290 may be acquired by using a different filter and the imaging sensor must have a sensitivity to the wavelengths of each of the plurality of filters. As an example and not by way of limitation, the HSI system of kitchen appliance 102 may scan wavelengths in the Y dimension, thereby allowing the system to generate 3D data cube 290 by compiling the captured spectral information over time to increase resolution in the Y dimension.
While the data demonstrated in
In particular embodiments, the hyperspectral imaging system of kitchen appliance 102 may compare the graph 408 of the spectrum of each region of interest 302, 402, 404, and 406 to one or more databases of reference spectra for object 220 identification. As an example and not by way of limitation, kitchen appliance 102 may retrieve previously recorded reference spectra of a plurality of objects 220 (e.g., foods) from memory 106 of kitchen appliance 102. As another example and not by way of limitation, kitchen appliance 102 may retrieve previously recorded reference spectra of the plurality of objects via network interface 116, wherein the reference spectra may be stored in one or more external datastores, databases, and/or other suitable storage method. In particular embodiments, one or more processors 104 of kitchen appliance 102 may instruct one or more algorithms of the hyperspectral imaging system to compare the measured spectrum of regions of interest against the database of reference spectra, wherein matches above a specified threshold may be sorted, weighed, and/or displayed to a user via the user interface of kitchen appliance 102.
In particular embodiments, one or more processors 104 of kitchen appliance 102 may instruct the illumination source 210 located within the interior of the kitchen appliance 102 to direct illumination on one or more objects 220. As an example and not by way of limitation, one or more sensors 108 of kitchen appliance 102 may capture a two-dimensional image 510 of objects 220 illuminated by illumination source 210. As an example and not by way of limitation, sensors 108 may include a monochromatic camera, and/or other suitable sensor 108. As another example and not by way of limitation, kitchen appliance 102 may capture a two-dimensional image 510 of objects 220 with one or more computer vision techniques.
In particular embodiments, the hyperspectral and/or multispectral imaging system of kitchen appliance 102 may compile a plurality of images taken by one or more sensors 108 of kitchen appliance 102 into three-dimensional data cube 290. As an example and not by way of limitation, one or more processors 104 of kitchen appliance 102 may instruct the imaging system to extract spatial and/or spectral information through slicing the 3D data cube 290 in one or more directions. In particular embodiments, after extracting spatial and/or spectral information from 3D data cube 290, kitchen appliance 102 may retrieve one or more spectral data sets in which two-dimensional image 510 may be spectrally compared to the spectral scale 520 of the previously stored data sets. As demonstrated by spectral scale 520, different colors and/or shades may correspond to individual objects 220. As an example and not by way of limitation, the hyperspectral and/or multispectral imaging system of kitchen appliance 102 may compare two-dimensional image 510 with the spectral scale 520 of one or more data sets stored locally on memory 108 of kitchen appliance 102 and/or stored externally in one or more external locations (e.g., separate server, cloud database) to generate an identification of objects 220. For example, as demonstrated in diagram 500, objects 220 may include a bell pepper, apple, avocado, and lime. Kitchen appliance 102 may then image the objects 220 through the techniques of hyperspectral imaging, multispectral imaging, and/or computer vision, either alone or in combination. As an example and not by way of limitation, one or more two dimensional images 510 may be taken, wherein the images may be compiled into the three-dimensional data cube 290. As another example and not by way of limitation, kitchen appliance 102 may extract spectral information from the 3D data cube 290 and match the spectral information with a database of previously stored spectral information corresponding to particular objects. For example, the spectral information may match the objects to their respective food types (e.g., bell pepper, apple, avocado, and lime).
In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to store each image containing spectrum information locally in memory 106 of kitchen appliance 102, wherein each image may further train one or more machine-learning based algorithms for identification and/or classification of objects 220. As another example and not by way of limitation, kitchen appliance 102 may instruct one or more processors 104 to store each image containing spectrum information externally in one or more external datastores, databases, and/or other suitable storage location. As an example and not by way of limitation, the database containing spectral information about a plurality of objects 220 (e.g., food items) may be continuously updated by kitchen appliance 102. In particular embodiments, one or more algorithms may be used to classify a plurality of objects 220 from a calibrated measurement of the objects spectrum as compared to the stored spectral information for a variety of objects. It is understood that the database may contain several different spectra corresponding to objects 220 to account for naturally occurring color differences (e.g., apples of different color). As an example and not by way of limitation, one or more algorithms such as least-squares fitting, principal component analysis, support vector machines (SVM), and/or neural network systems may be applied to the database of spectrum information. As another example and not by way of limitation, kitchen appliance 102 may instruct one or more processors 104 to perform one or more Computer-vision segmentation algorithms to assist with object identification.
In particular embodiments, objects 220 disposed inside the interior of kitchen appliance 102 may be contained inside packaging (e.g., a cake inside a cardboard box). For example, image 530 demonstrates foods (e.g., objects 220) located within packaging, wherein only a section of the surface of each food is visible to one or more sensors 108 of the hyperspectral imaging system. In particular embodiments, kitchen appliance 102 may use a machine-learning based computer vision technique to scan barcodes of packaging for object 220 identification. In particular embodiments, kitchen appliance 102 may utilize computer vision to generate a two-dimensional image 540 of the objects 220 within a packaging. As an example and not by way of limitation, kitchen appliance 102 may utilize both computer vision and hyperspectral and/or multispectral imaging to image objects 220. In this example, the hyperspectral and/or multispectral imaging system of kitchen appliance 102 may recognize foods by their spectral signature of a small region of interest, even if their total shape is occluded or obscured within the field of view of one or more sensors 108 (e.g., monochrome camera).
In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to execute instructions for automated cooking through hyperspectral and/or multispectral imaging-enabled monitoring of one or more objects 220. Temperature is one metric used for assessing objects 220 (e.g., food) during the cooking process. However, in many instances, such as baking or broiling, the doneness of food may be judged by changes in the food's visible color. In particular embodiments, kitchen appliance 102 may utilize HSI/MSI to provide additional information about the state of an object 220 during the cooking process, beyond what is obtained by a standard RGB camera. As an example and not by way of limitation, kitchen appliance 102 may utilize information obtained by HSI/MSI to predict the doneness of food being cooked, and further instructing kitchen appliance 102 to suspend the emission of radiation and/or heat into the cavity of the kitchen appliance 102 once the food is determined to be done cooking and preventing over-cooking.
In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to execute instructions to perform food quality monitoring through hyperspectral and/or multispectral imaging. In particular embodiments, controller 650 may instruct the imaging system of the electronic device (e.g., kitchen appliance 102) to capture subsequent sets of images of objects 220 over some period of time. As an example and not by way of limitation, kitchen appliance 102 may capture a second set of images of the objects 220, wherein the second set of images may be compared to the first set of images to determine a change in quality (e.g., food quality degradation). As an example and not by way of limitation, HSI/MSI may be used to identify adulterants in ingredients, assess fecal and/or bacterial contamination of foods (e.g., meats and produce), identify the presence of bone fragments in meat, identify defects (e.g., bruises) in produce, and quantify the ripeness of fruit. In general, identification of specific quality-correlated chemical signatures unique to particular foods may require large data sets, pre-calibration of the HSI/MSI apparatus, or intensive computation. Nonetheless, qualitative indicators of food quality change may be derived if the same food item is repeatedly imaged by HSI over a period of time. For example, a food storage kitchen appliance 102 (e.g., refrigerator) may instruct one or more processors 104 to conduct a longitudinal study of the same foodstuffs, offering the potential to track food quality over time and notify users of changes that may indicate spoilage. As an example and not by way of limitation, for particular categories of foods (e.g., unripe fruit), kitchen appliance 102 may utilize HSI/MSI to indicate an improvement in quality over time as the fruit ripens.
In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to perform food cataloging and/or tracking through the technique of HSI and/or MSI. In the case of a food-storage kitchen appliance 102 (e.g., refrigerator), the hyperspectral and/or multispectral imaging system may perform automatic recognition of food and allow for robust automatic inventory tracking with little to no user input required. Although some prepackaged food may be readily identified by their packaging via computer vision using RGB images, the addition of spectral data may extend the amount of information that may be gathered from each object 220 inside the cavity of kitchen appliance 102, resulting in a reduced error rate of identification of each object 220. As an example and not by way of limitation, kitchen appliance 102 utilizing HSI and/or MSI to identify one or more objects 220 disposed inside the interior may, in conjunction with other measurements (e.g., food weight as indicated by a strain gauge), track object type and quantity for the user, resulting in providing nutritional information to the user about what the user has cooked.
In particular embodiments, optical radiation may be emitted by a broadband radiation source disposed inside the interior of kitchen appliance 102, wherein each 2D image in a set of 2D images may represent a linear cross-section within the field of view of an imaging device, and wherein the second dimension of each 2D image may depict a variation of the linear-cross section across multiple wavelengths. As an example and not by way of limitation, an imaging process may be varied to select particular linear cross-sections within a field of view of the imaging device.
In particular embodiments, optical radiation may be emitted by one or more sources among a set of narrowband illumination sources disposed inside the interior of kitchen appliance 102, wherein each 2D image in the set of 2D images may be a 2D spatial view of the one or more objects imaged at a wavelength range corresponding to the optical radiation and wherein the optical radiation is varied to select particular wavelength ranges for sequential 2D images.
In particular embodiments, kitchen appliance 102 may contain one or more optical sensors suitable for capturing hyperspectral and/or multispectral data. In response to illumination by a broadband light source, as in step 610, the workflow may continue at step 630, wherein a monochrome camera may capture an image, and wherein a line in the image at a particular spatial position (e.g., X) may be selected. In this example, spectral information may be extracted using a diffraction grating and a monochromatic camera to generate a singular slice of the image along the transverse dimension (e.g., Y×λ). In particular embodiments, in response to illumination by a narrowband light source, as in step 620, the workflow may continue at step 640, wherein a monochromatic camera of kitchen appliance 102 may capture two-dimensional images of the interior of kitchen appliance 102 (e.g., X×Y) at a plurality of different wavelengths. In particular embodiments, kitchen appliance 102 may instruct one or more controllers 650 to combine the individual two-dimensional images from either of steps 630 or 640 into a 3D data cube 290.
At block 730, one or more controller(s) 650 may to send instructions to the hyperspectral imaging system of kitchen appliance 102 to align optics to select a line image 292. As an example and not by way of limitation, the one or more controllers 650 of kitchen appliance 102 may generate instructions for the hyperspectral imaging system of kitchen appliance 102 to align one or more components of the optical system (e.g., mirror 230, one or more objective lenses 240, slit 250, one or more collimating lenses 260, diffraction grating 270) with one or more sensors 108 of kitchen appliance 102 to generate a line image 292. In particular embodiments, at block 740, kitchen appliance 102 may instruct one or more controllers 650 to record one or more two-dimensional images via the one or more sensors 108 inside the interior of kitchen device 102. As an example and not by way of limitation, kitchen appliance 102 may record images in the Y-dimension as a function of wavelength. As another example and not by way of limitation, the hyperspectral imaging system of kitchen appliance 102 may be optically arranged so that one two-dimensional line image 292 in the Y-dimension may be recorded by the hyperspectral imaging system at a time, as the hyperspectral imaging system images object 220.
At block 750, kitchen appliance 102 may evaluate the plurality of two-dimensional line images 292 to determine that all rows and/or columns have been measured by one or more sensors 108 of kitchen appliance 102. In response to kitchen appliance 102 determining that all rows and/or columns have been measured by the one or more sensors 108, controller 650 may instruct one or more processors 104 of kitchen appliance 102 to construct a 3D data cube 290, as in block 760. In particular embodiments, kitchen appliance 102 may determine that all rows and/or columns have not been measured by one or more sensors 108 of kitchen appliance 102, in which case controller 650 may send instructions to kitchen appliance 102 to reperform the steps in block 73, 740, and 750.
In particular embodiments, at block 760, the hyperspectral imaging system of kitchen appliance 102 may construct a hyperspectral data cube by compiling the plurality of line scan images recorded at block 740 into a three-dimensional data cube 290. In particular embodiments, controller 650 of kitchen appliance may instruct one or more processors 104 to use the 3D data cube 290 constructed at block 760 to perform food identification and/or quality assessment of each object 220 (e.g., food) imaged. As an example and not by way of limitation, kitchen appliance 102, by extracting (x,y) subsets of the 3D data cube 290 at particular wavelengths, and/or by extracting wavelength subsets at a given (x,y) location, wherein images of objects 220 may be compared against a database of spectral information, wherein objects 220 may be identified. As another example and not by way of limitation, kitchen appliance 102 may instruct one or more processors 104 to execute instructions to repeat hyperspectral imaging and perform longitudinal quality monitoring of objects 220.
In particular embodiments, at block 840, the multispectral imaging system of kitchen appliance 102 may evaluate the two-dimensional images recorded in block 830 to determine that all LED wavelengths have been measured in the two-dimensional images. In response to kitchen appliance 102 determining that all LED wavelengths have been measured in the two-dimensional images, controller 650 may instruct one or more processors 104 of kitchen appliance 102 to construct a multispectral data cube, as in block 850. In particular embodiments, kitchen appliance 102 may determine that all LED wavelengths in the recorded two-dimensional images have not been measured, in which case controller 650 may send instructions to the multispectral imaging system of kitchen appliance 102 to reperform the steps in blocks 810, 820, and 830, and 840 until all LED wavelengths have been measured in the recorded two-dimensional images.
In particular embodiments, at block 850, the multispectral imaging system of kitchen appliance 102 may construct a 3D data cube 290 by compiling the plurality of recorded two-dimensional images in block 830 into a three-dimensional image. In particular embodiments, at block 860, controller 650 of kitchen appliance 102 may instruct one or more processors 104 of kitchen appliance 102 to perform food identification and/or quality assessment of each food imaged with the 3D data cube 290 as constructed in block 850. In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to store each image containing spectrum information locally in memory 106 of kitchen appliance 102, wherein each image may further train one or more machine-learning based algorithms for identification and/or classification of objects 220. As another example and not by way of limitation, kitchen appliance 102 may instruct controller 650 to store each image containing spectrum information externally in one or more external datastores, databases, and/or other suitable storage location. As an example and not by way of limitation, the database containing spectral information about a plurality of objects 220 (e.g., food items) may be continuously updated by kitchen appliance 102. In particular embodiments, one or more algorithms may be used to classify a plurality of objects 220 from a calibrated measurement of the objects spectrum as compared to the stored spectral information for a variety of objects. As an example and not by way of limitation, one or more algorithms such as least-squares fitting, principal component analysis, support vector machines (SVM), and/or neural network systems may be applied to the database of spectrum information. As another example and not by way of limitation, kitchen appliance 102 may instruction one or more processors 104 to perform one or more computer-vision segmentation algorithms for object identification.
In particular embodiments, monochrome camera 920 may be communicatively coupled to controller 650. As an example and not by way of limitation, monochrome camera 920 may be placed in the interior of kitchen appliance 102 an capture two-dimensional images of objects 220 illuminated by single wavelength illumination generated by LED array 910. In particular embodiments, controller 650 may generate instructions to the multispectral imaging system of kitchen appliance 102 to generate a controlled illumination sequence into the interior of kitchen appliance 102 by LED array 910.
In particular embodiments, after monochrome camera 920 captures one or more two-dimensional images, controller 650 may instruct the multispectral imaging system of kitchen appliance 102 to step through a plurality of illumination wavelengths and record images (e.g., X×Y images) at each wavelength to construct a 3D data cube. In particular embodiments, as displayed by diagram 900, at block 930, controller 650 may instruct the multispectral imaging system of kitchen appliance 102 to construct a 3D data cube 290 (e.g., multispectral data cube) by compiling the plurality of images captured by monochrome camera at block 920. In particular embodiments, after constructing the 3D data cube 290, at block 940, controller 650 may instruct the multispectral imaging system of kitchen appliance 102 to perform food segmentation and/or food identification. As an example and not by way of limitation, controller 650 may use the image data, either alone or in combination with one or more computer vision approaches to segment the individual food items (e.g., objects 220) and classify the food according to one or more food types.
In particular embodiments, controller 650 may instruct the imaging system of kitchen appliance 102 to save the segmented and identified food items, as identified in block 940, to a food inventory 220 stored in the kitchen appliance 102 at block 950. As an example and not by way of limitation, the food may not be identified by the imaging system of kitchen appliance 102, in which case controller 650 may instruct the imaging system to repeat measurements and generate updated images of the objects 220. As an example and not by way of limitation, the database of food items may be stored locally in memory 106 of kitchen appliance 102. As another example and not by way of limitation, the database of food items may be stored externally at one or more servers, wherein one or more processors 104 of kitchen appliance 102 may instruct network interface 116 to retrieve the database of food items from one or more external locations (e.g., remote server(s), cloud storage database(s)). For example, the imaging system of kitchen appliance 102 may save the identified individual food items located in the interior of kitchen appliance 102 in order to respond to one or more queries by a user.
In particular embodiments, at block 960, controller 650 may instruct the imaging system of kitchen appliance 102 to repeat the measurements at blocks 930, 940, and 950 indefinitely. For each food item identified, key spectral indicators may be stored and tracked over time. As an example and not by way of limitation, as the imaging system of kitchen appliance 102 repeats measurements, the user may be informed if one or more individual food items deviate beyond a particular threshold, which may indicate a substantial change in quality of the food items.
In particular embodiments, at block 1008, the multispectral imaging system of kitchen appliance 102 may evaluate the one or more two-dimensional images recorded at block 1006 to determine that all LED wavelengths have been measured in the two-dimensional image(s). In response to determining that all LED wavelengths have been measured in the recorded two-dimensional images, controller 650 may instruct one or more processors 104 of kitchen appliance 10 to construct a 3D data cube 290, as in block 1010. In particular embodiments, the multispectral imaging system of kitchen appliance 102 may determine that all LED wavelengths have not been measured, in which case controller 650 may send instructions to the multispectral imaging system of kitchen appliance 102 to reperform the steps of block 1004, 1006, and 1008 until all LED wavelengths have been measured in the recorded two-dimensional images.
In particular embodiments, after the 3D data cube 290 is constructed, as discussed in block 1010, controller 650 may instruct the multispectral imaging system of kitchen appliance 102 to perform image segmentation, as in block 1012. In particular embodiments, controller 650 of the multispectral imaging system of kitchen appliance 102 may perform food identification, as in block 1014. As an example and not by way of limitation, the segmented images of block 1012 may be compared to a database consisting of food spectrum data for food identification. For example, the segmented image of a red apple (e.g., object 220) may be compared against the database of food spectrum data, wherein the multispectral imaging system may compare the images to a plurality of images with similar spectrum data until the system reaches a confidence value above a certain threshold of object 220 identification.
In particular embodiments, in response to the food identification in block 1014, the multispectral imaging system of kitchen appliance 102 may determine the food item presently exists in a record of the food item inventory of kitchen appliance 102. As an example and not by way of limitation, the record of food item inventory may be stored locally in memory 106 of kitchen appliance 102. As another example and not by way of limitation, the record of food item inventory may be stored externally by one or more separate servers. For example, the multispectral imaging system of kitchen appliance 102 may determine that the red apple has been previously stored in the food item inventory of kitchen appliance 102, and will not add the identified red apple to the inventory record. In an alternate example, the multispectral imaging system of kitchen appliance 102 may determine that an avocado has not been previously stored in the food item inventory of kitchen appliance 102, thereby indicating to the multispectral imaging system that the avocado is a new item and subsequently adds the avocado to the food item inventory. In particular embodiments, in response to the food identification in block 1014, the multispectral imaging system of kitchen appliance 102 may determine the food item does not presently exist in a record of the food item inventory of kitchen appliance 102, in which case the multispectral imaging system may add the identified food item to an inventory list at block 1018. As an example and not by way of limitation, the food inventory list may be presented to a user via a user interface of kitchen appliance 102. In this example, the user may interact with the food inventory list through one or more selectable options presented by the user interface of kitchen appliance 102.
In particular embodiments, in response to the multispectral imaging system of kitchen appliance 102 determining that the identified foods in block 1040 are presently recorded in a food inventory list, controller 650 may instruct the multispectral imaging system to compare the most recent set of recorded two-dimensional images to an original measurement at block 1018. In particular embodiments, in response to determining the food item presently exists in the food inventory list, controller 650 may instruct the multispectral imaging system to repeat measurements indefinitely. As an example and not by way of limitation, controller 650 may instruct the multispectral imaging system to reperform the steps in blocks 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, and 1018 until achieving a termination criteria. In this example, a termination criterion may be removal of the food item from inventory. As an example and not by way of limitation, if the multispectral imaging system of kitchen appliance 102 determines that the red apple is already in the food item inventory of kitchen appliance 102, the multispectral imaging system may reperform the image capture process indefinitely, and after a few days the multispectral imaging system may detect a change in the quality of the red apple. In this example, the kitchen appliance 102 may provide a notification to a user via the user interface of kitchen appliance 102 containing information that the quality of the red apple is decreasing (e.g., the apple is rotting). As another example and not by way of limitation, kitchen appliance 102 may generate recipe suggestions based upon the identified foods within the interior of the appliance. For example, kitchen appliance 102 may present a recipe suggestion to a user via a user interface of the kitchen appliance 102, wherein the recipe uses raw food ingredients determined to soon suffer a degradation in quality. For example, if vegetables in a user's refrigerator are soon to expire, the kitchen appliance 102 may recommend a salad recipe to the user via a user interface of kitchen appliance 102.
In particular embodiments, after line scan optics in block 1120 and camera 1130 capture two-dimensional images, controller 650 may instruct the hyperspectral imaging system of kitchen appliance 102 to sweep the line image through the scene, recording two-dimensional images (e.g., Y×λ) for each line. As an example and not by way of limitation, controller 650 may perform optomechanical control of the line scan optics in block 1120 for scanning line-images for capture by the camera 1130. In particular embodiments, at block 1140, controller 650 may instruct the hyperspectral imaging system of kitchen appliance 102 to construct a 3D data cube 290 by compiling the plurality of images captured by the line-scan optics and camera 1130. In particular embodiments, after constructing the 3D data cube 290 at block 1140, controller 650 may instruct the hyperspectral imaging system of kitchen appliance 102 to perform food segmentation and/or identification at block 1150. As an example and not by way of limitation, controller 650 may use the image data, either alone or in combination with one or more computer vision approaches to segment the individual food items (e.g., objects 220) and classify the food according to one or more food types. The segmentation and identification of individual food items may allow kitchen appliance 102 to recommend a cooking schedule to the user based on the foods identified by the hyperspectral imaging system. For example, if the user places potatoes within the interior of a microwave oven (e.g., kitchen appliance 102), the hyperspectral imaging system of kitchen appliance 102 may identify the object 220 as potatoes and subsequently recommend one or more cooking schedules to the user based on the identified foods within the interior of the microwave oven.
In particular embodiments, after performing food segmentation and/or identification as in block 1150, kitchen appliance 102 may recommend a recipe to the user via a user interface of kitchen appliance 102 as in block 1160. As an example and not by way of limitation, a user may select the recommended recipe via the user interface of kitchen appliance 102, in which case kitchen appliance 102 may perform temperature scheduling and/or the cooking process in accordance with the selected recipe. As another example and not by way of limitation, the HSI system of kitchen appliance 102 may image foods to perform an automatic analysis of suggested cooking time between based on particular qualities of the foods. For example, the cook time may differ for bone-in versus boneless cuts of meat, as well as relative fat and muscle content of the meat. In this way, kitchen appliance 102 may adjust the temperature and cooking time based on the cut of meat.
In particular embodiments, at block 1170, after performing food segmentation and/or identification as in block 1150, controller 650 may instruct the hyperspectral imaging system of kitchen appliance 102 to repeat measurements to monitor food doneness over the duration of the cooking process. As an example and not by way of limitation during the cooking process, controller 650 may monitor the spectra of selected foods for changes over time that indicate doneness. As another example and not by way of limitation, in response to identifying markers of food doneness, controller 650 may instruct one or more heating elements within kitchen appliance 102 to terminate the heating process.
In particular embodiments, at block 1208, controller 650 of the hyperspectral imaging system of kitchen appliance 102 may evaluate the one or more two-dimensional images recorded at block 1206 to determine that all lines have been measured. In response to controller 650 determining all lines have been measured in block 1208, controller 650 may instruct the hyperspectral imaging system to construct a hyperspectral data cube, as in block 1210. In particular embodiments, at block 1208, controller 650 of the hyperspectral imaging system may determine that all lines have not been measured, in which case controller 650 may send instructions to the hyperspectral imaging system to repeat the steps of block 1204, 1206, and 1208 until all lines have been measured.
At block 1212 the controller 650 of the hyperspectral imaging system may perform image segmentation of the recorded two-dimensional images. Next, at block 1214, controller 650 of the hyperspectral imaging system may perform food identification by comparing the segmented images against a database consisting of food spectrum data. As an example and not by way of limitation, the hyperspectral imaging system of kitchen appliance 102 may store a database of known food items locally in memory 106, or externally on one or more separate servers and/or cloud databases.
In particular embodiments, after the food is identified by the hyperspectral imaging system at block 1214, the controller 650 may generate instructions to suggest a recipe to a user via a user interface of kitchen appliance 102, as in block 1216. In particular embodiments, in response to receiving user input via a user interface of kitchen appliance 102, controller 650 may instruct one or more heating elements within kitchen appliance 102 to emit radiation into the interior of the kitchen appliance 102, thereby beginning the cooking process in block 1218. At block 1220, previously stored food spectrum data may be compared to the real-time two-dimensional images recorded by one or more sensors 108 of kitchen appliance 102. At block 1222, controller 650 of the hyperspectral imaging system may determine the food is done cooking through a comparison of previously stored cooking data of cooked food and real time two-dimensional images recorded by a camera in block 1206. In particular embodiments, the hyperspectral imaging system of kitchen appliance 102 may determine the food in the interior of kitchen appliance 102 is done cooking, wherein controller 650 may send instructions to one or more heat emitters of kitchen appliance 102 to terminate the emission of heat and inform the user that the cooking process is complete via a user interface of kitchen appliance 102, as in block 1224. In particular embodiments, controller 650 may determine that the food within the interior of kitchen appliance 102 is not finished cooking, in which case controller 650 may instruct the hyperspectral imaging system to reperform the steps of blocks 1202, 1204, 106, 1208, 1210, 1212, 1214, 1216, 1218, 1220, and 1222 until controller 650 determines the food is done cooking.
The method 1300 may begin at block 1310 with the electronic device (e.g., kitchen appliance 102) emitting optical radiation (e.g., spectral illumination) to illuminate one or more first objects disposed inside an interior of the electronic device. As an example and not by way of limitation, the optical radiation may be emitted by one or more radiation sources located within the interior of kitchen appliance 102. As another example and not by way of limitation, optical radiation may include a broad band illumination source, and narrow-band illumination source, a broad spectrum LED, a single wavelength LED, LED array, or any suitable illumination source 210.
The method 1300 may then continue at block 1320 where the electronic device (e.g., kitchen appliance 102) may capture a first set of 2D images of the one or more first objects illuminated by the optical radiation, wherein the variation of illumination and/or variation of the imaging process permits the first set of 2D images to be combined into a representation of the one or more objects as imaged at multiple wavelengths. As an example and not by way of limitation, each image of the first set of images may portray the one or more first objects at one or more recorded wavelengths. As another example and not by way of limitation, each image of the first set of images may alternatively portray the one or more line scans of a one-dimensional slice of the scene, with the other dimension of the image representing a variation of that line scan across different wavelengths. The first set of images of the one or more first objects illuminated by the optical radiation may be captured by one or more sensor(s) 108 of kitchen appliance 102, including but not limited to a camera, monochrome camera, or any suitable sensor.
The method 1300 may then continue at block 1330 with the electronic device (e.g., kitchen appliance 102) determining whether the first set of images comprises a representation of the one or more objects as imaged at a sufficient number of total recorded wavelengths. In particular embodiments, method 1300 may continue at block 1340, wherein in response to determining that the first set of images comprises a representation of the one or more objects as imaged at a sufficient number of total recorded wavelengths, generating a three-dimensional (3D) spectral data cube representing images of the of the one or more first objects in two spatial dimensions at multiple recorded wavelengths. As an example and not by way of limitation, the first set of images may be compiled to generate the spectral map (e.g., 3D data cube 290) of the one or more first objects.
The method 1300 may then continue at block 1350 with the electronic device (e.g., kitchen appliance 102) storing the 3D spectral map (e.g., 3D data cube 290) for processing by the electronic device. As an example and not by way of limitation, the kitchen appliance 102 may store the 3D spectral map locally in memory 106 of kitchen appliance 102. As another example and not by way of limitation, kitchen appliance 102 may store the 3D spectral map externally in one or more databases, datastores, and/or other suitable external storage location accessible via network interface 116.
In particular embodiments, controller 650 may instruct the hyperspectral and/or multispectral imaging systems to retrieve stored 3D spectral maps (e.g., 3D spectral data cube 290) of the one or more first objects and determine one or more similarities between the 3D spectral data cube 290 of the first objects and one or more previously stored 3D spectral data cubes 290 of one or more second objects of one or more object types. Controller 650 may then classify one or more of the first objects based on the determined similarities between the one or more first objects and the previously stored 3D spectral data cubes of one or more second objects.
In particular embodiments, the hyperspectral and/or multispectral imaging systems of kitchen appliance 102 may calculate a probability score through one or more machine-learning models that each of the first objects are associated with a particular object type. As an example and not by way of limitation, based on the comparison between the 3D spectral map of the first sequence of the plurality of images and the previously stored 3D spectral maps of a plurality of object types, one or more processors 104 of the hyperspectral and/or multispectral imaging systems of kitchen appliance 102 may calculate, through one or more machine-learning systems, a probability score that the first object type is associated with a particular object type for each of the one or more first objects. In particular embodiments, the hyperspectral and/or multispectral imaging systems of kitchen appliance 102 may compare the probability score of each of the one or more first objects against a threshold score. As an example and not by way of limitation, the threshold may be predetermined and stored locally in the kitchen appliance 102 or, alternatively, stored externally by one or more external servers. In particular embodiments, kitchen appliance 102 may determine the probability score associated with the identification of the particular object is below or fails to meet the threshold, in which case controller 650 may instruct the hyperspectral and/or multispectral imaging systems to repeat measurements and capture updated sets of images of the one or more objects 220 illuminated by spectral illumination.
In particular embodiments, each of the first object with a probability score greater than the threshold score may be classified as being of the respective particular object type and store the identification of the particular object (e.g., the object is an avocado) locally (e.g., in memory 108) or externally in an inventory. As an example and not by way of limitation, each first object classified as being of the respective particular object type may be added to an inventory of the one or more first objects disposed inside the interior of the kitchen appliance 102. As another example and not by way of limitation, one or more machine-learning systems of kitchen appliance 102 may generate recommendations to be displayed on a user interface of the kitchen appliance 102 based on the inventory of the classified one or more objects disposed inside the interior of the electronic device. For example, the recommendation may be a recipe that includes ingredients identified within the interior of the kitchen appliance 102. As another example, the kitchen appliance 102 may generate a recommendation that a user consume a particular object within a period of time, before the quality of the object decreases. In this example, the kitchen appliance 102 (e.g., refrigerator) may, for example, suggest that a user consume the salmon before it spoils. Kitchen appliance 102 may determine a change in quality of the one or more first objects by capturing a second set of images of the one or more first objects illuminated by the spectral illumination, wherein a comparison between the first and second set of images may reflect a change in quality of the one or more objects.
In particular embodiments, at block 1540, the multispectral imaging system of kitchen appliance 102 may evaluate the two-dimensional images recorded in block 1530 to perform a determination that all LED wavelengths have been measured in the two-dimensional images, as in block 1540. In response to kitchen appliance 102 determining that all LED wavelengths have been measured in the two-dimensional images as in block 1540, controller 650 may instruct one or more processors 104 of kitchen appliance 102 to construct a 3D data cube 290, as in block 1550. In particular embodiments, kitchen appliance 102 may determine that all LED wavelengths in the recorded two-dimensional images have not been measured, in which case controller 650 may send instructions to the multispectral imaging system of kitchen appliance 102 to reperform the steps in blocks 1510, 1520, and 1530, and 1540 until all LED wavelengths have been measured in the recorded two-dimensional images.
In particular embodiments, at block 1550, the hyperspectral imaging system of kitchen appliance 102 may construct a 3D data cube 290 by compiling the plurality of recorded two-dimensional images in block 1530 into a three-dimensional image. In particular embodiments, at block 1560, controller 650 of kitchen appliance 102 may instruct one or more processors 104 of kitchen appliance 102 to perform food identification and/or quality assessment of each food (e.g., object 220) imaged with the 3D data cube 290, as constructed in block 1550. In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to store each image containing spectrum information locally in memory 106 of kitchen appliance 102, wherein each image may further train one or more machine-learning based algorithms for identification and/or classification of objects 220. As another example and not by way of limitation, kitchen appliance 102 may instruct controller 650 to store each image containing spectrum information externally in one or more external datastores, databases, and/or other suitable storage location. As an example and not by way of limitation, the database containing spectral information about a plurality of objects 220 (e.g., food items) may be continuously updated by kitchen appliance 102. In particular embodiments, one or more algorithms may be used to classify a plurality of objects 220 from a calibrated measurement of the objects spectrum as compared to the stored spectral information for a variety of objects. As an example and not by way of limitation, one or more algorithms such as least-squared fitting, principal component analysis, support vector machines (SVM), and/or neural network systems may be applied to the database of spectrum information. As another example and not by way of limitation, kitchen appliance 102 may instruction one or more processors 104 to perform one or more computer-vision segmentation algorithms for object identification.
In particular embodiments, camera 1130 may capture one or more two-dimensional images under single wavelength illumination. In particular embodiments, controller 650 may instruct the imaging system of kitchen appliance 102 to construct a 3D data cube 290 by compiling the plurality of two-dimensional images captured by camera 1130. In particular embodiments, controller 650 may use the image data 3D data cube 290, either alone or in combination with one or more computer vision approaches to segment the individual food items (e.g., objects 220), and perform food discrimination and/or quality assessment 1640. In particular embodiments, controller 650 may combine each 2D image captured by camera 1130 into a multispectral data cube, wherein the imaging system may perform object segmentation to isolate individual objects 220. In this example, the spectral signature of each item 220 may be used to identify the object 220, and perform food discrimination and/or quality assessment as in step 1640. Step 1640 may be performed by the imaging system in combination with computer vision approaches based on visible-light images.
In particular embodiments, at block 1740, the imaging system of kitchen appliance 102 may evaluate the two-dimensional images recorded in block 1730 to determine that all wavelengths have been measured in the two-dimensional images. In response to kitchen appliance 102 determining that all LED wavelengths have been measured in the two-dimensional images, controller 650 may instruct one or more processors 104 of kitchen appliance 102 to construct a 3D data cube 290, as in block 1750. In particular embodiments, kitchen appliance 102 may determine that all LED wavelengths in the recorded two-dimensional images have not been measured, in which case controller 650 may send instructions to the multispectral imaging system of kitchen appliance 102 to repeat the steps in blocks 1710, 1720, and 1730, and 1740 until all LED wavelengths have been measured in the recorded two-dimensional images.
In particular embodiments, at block 1750, the imaging system of kitchen appliance 102 may construct a 3D data cube 290 by compiling the plurality of recorded two-dimensional images in block 1730 into a three-dimensional image. In particular embodiments, at block 1760, controller 650 of kitchen appliance 102 may instruct one or more processors 104 of kitchen appliance 102 to perform food identification and/or quality assessment of each food imaged with the 3D data cube 290 as constructed in block 1750. In particular embodiments, kitchen appliance 102 may instruct one or more processors 104 to store each image containing spectrum information locally in memory 106 of kitchen appliance 102, wherein each image may further train one or more machine-learning based algorithms for identification and/or classification of objects 220.
Systems and Methods
This disclosure contemplates any suitable number of computer systems 1800. This disclosure contemplates computer system 1800 taking any suitable physical form. As example and not by way of limitation, computer system 1800 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system 1800 may include one or more computer systems 1800; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks.
Where appropriate, one or more computer systems 1800 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example, and not by way of limitation, one or more computer systems 1800 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 1800 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
In particular embodiments, computer system 1800 includes a processor 1802, memory 1804, storage 1806, an input/output (I/O) interface 1808, a communication interface 1810, and a bus 1812. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement. In particular embodiments, processor 1802 includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, processor 1802 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1804, or storage 1806; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 1804, or storage 1806. In particular embodiments, processor 1802 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 1802 including any suitable number of any suitable internal caches, where appropriate. As an example, and not by way of limitation, processor 1802 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 1804 or storage 1806, and the instruction caches may speed up retrieval of those instructions by processor 1802.
Data in the data caches may be copies of data in memory 1804 or storage 1806 for instructions executing at processor 1802 to operate on; the results of previous instructions executed at processor 1802 for access by subsequent instructions executing at processor 1802 or for writing to memory 1804 or storage 1806; or other suitable data. The data caches may speed up read or write operations by processor 1802. The TLBs may speed up virtual-address translation for processor 1802. In particular embodiments, processor 1802 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 1802 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 1802 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1802. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
In particular embodiments, memory 1804 includes main memory for storing instructions for processor 1802 to execute or data for processor 1802 to operate on. As an example, and not by way of limitation, computer system 1800 may load instructions from storage 1806 or another source (such as, for example, another computer system 1800) to memory 1804. Processor 1802 may then load the instructions from memory 1804 to an internal register or internal cache. To execute the instructions, processor 1802 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 1802 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 1802 may then write one or more of those results to memory 1804. In particular embodiments, processor 1802 executes only instructions in one or more internal registers or internal caches or in memory 1804 (as opposed to storage 1806 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 1804 (as opposed to storage 1806 or elsewhere).
One or more memory buses (which may each include an address bus and a data bus) may couple processor 1802 to memory 1804. Bus 1812 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 1802 and memory 1804 and facilitate accesses to memory 1804 requested by processor 1802. In particular embodiments, memory 1804 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 1804 may include one or more memory devices 1804, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.
In particular embodiments, storage 1806 includes mass storage for data or instructions. As an example, and not by way of limitation, storage 1806 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 1806 may include removable or non-removable (or fixed) media, where appropriate. Storage 1806 may be internal or external to computer system 1800, where appropriate. In particular embodiments, storage 1806 is non-volatile, solid-state memory. In particular embodiments, storage 1806 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 1806 taking any suitable physical form. Storage 1806 may include one or more storage control units facilitating communication between processor 1802 and storage 1806, where appropriate. Where appropriate, storage 1806 may include one or more storages 1806. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.
In particular embodiments, I/O interface 1808 includes hardware, software, or both, providing one or more interfaces for communication between computer system 1800 and one or more I/O devices. Computer system 1800 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 1800. As an example, and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 1806 for them. Where appropriate, I/O interface 1808 may include one or more device or software drivers enabling processor 1802 to drive one or more of these I/O devices. I/O interface 1808 may include one or more I/O interfaces 1806, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.
In particular embodiments, communication interface 1810 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 1800 and one or more other computer systems 1800 or one or more networks. As an example, and not by way of limitation, communication interface 1810 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 1810 for it.
As an example, and not by way of limitation, computer system 1800 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 1800 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 1800 may include any suitable communication interface 1810 for any of these networks, where appropriate. Communication interface 1810 may include one or more communication interfaces 1810, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.
In particular embodiments, bus 1812 includes hardware, software, or both coupling components of computer system 1800 to each other. As an example, and not by way of limitation, bus 1812 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 1812 may include one or more buses 1812, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.
Miscellaneous
Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.
Herein, “automatically” and its derivatives means “without human intervention,” unless expressly indicated otherwise or indicated otherwise by context.
The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.
The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10853758 | Schutt | Dec 2020 | B2 |
| 10935310 | Junge | Mar 2021 | B2 |
| 20180042483 | Bardhan | Feb 2018 | A1 |
| 20180239319 | Abdoo | Aug 2018 | A1 |
| 20190200797 | Diao | Jul 2019 | A1 |
| 20200184624 | Jang | Jun 2020 | A1 |
| 20210089769 | Ryu | Mar 2021 | A1 |
| 20210096027 | Nunez Sanchez | Apr 2021 | A1 |
| 20210174495 | Nipe | Jun 2021 | A1 |
| 20210228022 | Liu | Jul 2021 | A1 |
| 20220210347 | Noh | Jun 2022 | A1 |
| 20220252451 | Kim | Aug 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 2020-178052 | Sep 2020 | WO |
| Entry |
|---|
| “Smart Oven: 12-1 Table Top Oven”, by June Oven; https://juneoven.com/pages/smart-oven, 2021. |
| “Keep track of what's in your smart fridge with Family Hub”, by Samsung; https://www.samsung.com/us/support/answer/ANS00049761/, 2022. |
| International Search Report and Written Opinion in International Application No. PCT/KR2023/007939, dated Sep. 15, 2023. |
| Number | Date | Country | |
|---|---|---|---|
| 20240029337 A1 | Jan 2024 | US |
