Patent Grant
12082742
This invention relates to kitchen appliances and more particularly to robotic kitchen appliances capable of performing a wide range of food preparation steps in a restaurant environment.
Providing a robust and effective apparatus or combination of apparatuses to prepare food for consumers is challenging because of the wide variety of types of food, cooking techniques, kitchen appliances, kitchen tools, and utensils. Additionally, food preparation is often labor intensive and subject to human error. Workers employed by these businesses require careful and sometimes excessive training to accurately and safely prepare the food, thus increasing costs. It follows that businesses that prepare and sell food typically have high labor costs and experience large amounts of monetary and food loss as well as customer dissatisfaction due to human error.
Various commercial food preparation equipment addressed some of these challenges. The existing equipment, however, has several drawbacks. First, food preparation equipment is usually designed as a bespoke solution to perform a limited scope of work. Chaining together many different pieces into a workflow is a complex and expensive process and results in a system with highly limited capabilities yet a large footprint. Second, such food preparation equipment typically requires batch preparation of food items. For example, clamshell grills are typically designed to prepare multiple food items in batches and are not useful for preparing items individually. Third, the increased mechanical and/or electrical complexity inherent in such devices often leads to increased failure rates versus conventional, non-automated versions of such devices, resulting in higher downtimes. Such downtimes can be especially costly for restaurants because restaurants do not typically have back-up equipment onsite and consequently they may not be able to cook a number of items on their menu which reduces average order size or even drives away potential customers. Fourth, such food preparation equipment typically has a large footprint compared to conventional versions of the equipment and for the variety of items they are capable of cooking. This larger size is a challenge for restaurant owners because of the high cost of kitchen space. For example, the MasterMatic fryer is substantially larger than a similar model without the automatic conveyor. Fifth, the potential benefits of such food preparation equipment are often outweighed by their associated upfront costs. For example, automated frying equipment is significantly more expensive than conventional frying equipment. Sixth, such food preparation equipment still requires extensive involvement of kitchen workers. Seventh, most food preparation equipment doesn't interact with outside data to achieve optimal production scheduling and reduce food wasted. Because of the above challenges, use of automation in restaurant and food service operations is generally limited to spot solutions and does not take advantage of the availability of data to build accurate demand models and then use these models to automatically feed a production schedule.
Additionally, there are challenges associated with utensils for frying food items. Fry baskets are used to contain food during the cooking process in a fryer, enabling easy extraction from product. An alternative approach is termed “open-bay” cooking, where food is tossed in the fryer and then shoveled out. Baskets are traditionally made from wire or sheet metal with silicone handles to provide thermal insulation for a kitchen worker grabbing the basket. Despite the insulated handle, the worker is exposed to hot oil and hot equipment as well as the risk of slipping and falling arising from oil splatter in the vicinity of the fryer.
Accordingly, a robotic kitchen system and method that overcomes the above-mentioned challenges is desirable.
A robotic kitchen system for preparing food items in combination with at least one kitchen appliance in a commercial or restaurant kitchen comprises a robotic arm and end effector or tool, a basket held by the end effector, and an automated bin assembly in the vicinity of the robotic arm. The automated bin assembly includes a number of bins for holding the food items. A camera or sensor array is aimed towards the bins for collecting image data of the food items in the bin. A central processor is operable to compute and provide directions to the first robotic arm and automated bin assembly to (a) move the robotic arm to the bin, and (b) actuate the bin to drop the food items from the bin into the basket.
In embodiments, the central processor is further operable to compute and provide directions to the robotic arm to move the basket into a fryer after the basket receives the food items.
In embodiments, the robotic kitchen system further comprises a shield and first window or opening in the shield through which at least a portion of the bin can protrude to receive the food items from outside the shield. In embodiments, the bin is operable to rotate from a first position in which a front portion of the bin protrudes through the first window, and a second position in which the food items fall out a rear opening of the bin.
In embodiments, the central processor is further operable to compute and provide directions to transfer the food items through a second window after the food items are cooked in the fryer.
In embodiments, the robotic kitchen system further comprises a ramp or chute extending through the second window, and arranged at a downward angle such that the food items placed on the ramp slide down the ramp out the second window.
In embodiments, the automated bin status comprises a plurality of bins, each of which is automatically rotatable.
In embodiments, the robotic kitchen system comprises a plurality of fryers.
In embodiments, the robotic kitchen system further comprises a scheduling engine to determine a sequence of food preparation steps to detect and transfer the food items from each of the bins to the fryers for cooking based on a plurality of inputs selected from the group consisting of camera data, customer orders, inventory, and recipe information.
In embodiments, the robotic kitchen system further comprises a frame, to which the shield is attached, and a linear guide to which the robotic arm is movably coupled.
In embodiments, a method of robotically preparing food items in a commercial or restaurant kitchen having at least one kitchen appliance for cooking a food item includes the following steps: detecting the food item placed in a bin; robotically manipulating a basket underneath the bin, wherein the robotically manipulating is performed with a robotic arm; and automatically moving the bin to transfer the food items to the basket.
In embodiments, the method further comprises robotically manipulating the basket into the cooking appliance after the food item has been transferred to the basket.
In embodiments, the method further comprises classifying, using a programmed processor and image data, the food item in the bin.
In embodiments, the method further comprises determining, using a programmed processor, a cook schedule for the food item based on type of food item.
In embodiments, the method further comprises removing the basket containing the food item from the cooking appliance after the food item has been cooked.
In embodiments, the method further comprises robotically dumping the cooked food item onto a target surface. In embodiments, the target surface is a ramp, and the ramp is shaped and arranged such that the food item dumped thereon slides downward and falls into a food holding area.
In embodiments, the method further comprises providing a shield in front of the robotic arm and fryer, and wherein the shield comprises a first opening for a portion of the bin to protrude.
In embodiments, the step of automatically moving the bin is performed by rotating the bin using a motor or actuator removably coupled to the bin.
In embodiments, the method further comprises decoupling the bin from the motor and cleaning the bin.
In embodiments, the method further comprises controlling, using a programed processor, the robotic arm to robotically manipulate the basket based on image data from a camera or sensor.
In embodiments, the method further comprises controlling, using a programed processor, the bin to transfer the food item to the basket based on image data from a camera or sensor.
An automated food bin station for transferring food items from a human to robotic kitchen assistant comprises: a bin; a motor removably coupled to the bin and operable to move the bin between a first position for receiving the food items and to a second position for dumping the food items; a sensor array for obtaining image data of the bin; a processor for detecting and classifying food items in the bin, and for instructing the motor to move the bin between the first position and the second position.
In embodiments, the automated bin station further comprises a ramp extending through the second window, and arranged at a downward angle such that the food items placed on the ramp from within the enclosure slide down the ramp out the second window.
Optionally, a sorting bin for collecting the food from the ramp is arranged below the egress end of the ramp. In embodiments, a linear rail and motor are adapted to move the sorting bin laterally and rotationally and the central processor is further operable to compute and provide directions to move the sorting bin to at least one food holding tray, and to dump the food into the food holding tray.
In a work environment requiring a human to collaboratively work with a robot, a method for securing a robot workspace from a human workspace to protect the human from injury arising from contact with the robot comprises automatically presenting an object collection zone from the robot workspace through a window in a physical barrier into the human workspace, said physical barrier separating the human workspace from the robot workspace; and automatically manipulating the object collection zone to transport an object placed in the object collection zone to the robot workspace, wherein the manipulating step is performed independent of the human and robot.
In embodiments, the method further comprises automatically detecting the object when the object is placed in the object collection zone. An example of the object is a food item such as frozen fries or chicken fingers. Examples of the object collection zone include, without limitation, a bin, chamber, container, and cart as well as space provided on a conveyor belt assembly, rack, or guide system.
In embodiments, the method further comprises positioning a cooking tool underneath a portion of the object collection zone, wherein the positioning is performed by the robot.
In embodiments, the method further comprises automatically moving the object collection zone to transfer the object to the cooking tool. An example of a cooking tool is a fry basket.
In embodiments, the method further comprises robotically moving the cooking tool to dump the object onto a pathway, wherein the pathway is configured to transport the object from inside the robot workspace to outside the robot workspace. In embodiments, the pathway is a chute, and the chute is arranged with a downward sloping angle to transport the object based on gravity.
In embodiments, the step of automatically manipulating is performed by an automated bin assembly comprising a plurality of computer controlled bins, each of which is adapted to rotate or tilt from a first orientation to receive the object from the human in the human workspace, to a second orientation to position the object inside the robot workspace.
An automated sorting station as described herein.
An automated bin station as described herein.
A method for transferring food items from a bin to a basket as described herein.
A robotic kitchen system operable to transfer food items from a bin to a basket as described herein.
The description, objects and advantages of embodiments of the present invention will become apparent from the detailed description to follow, together with the accompanying drawings.
Before the present invention is described in detail, it is to be understood that this invention is not limited to particular variations set forth herein as various changes or modifications may be made to the invention described and equivalents may be substituted without departing from the spirit and scope of the invention. As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act(s) or step(s) to the objective(s), spirit or scope of the present invention.
Methods recited herein may be carried out in any order of the recited events which is logically possible, as well as the recited order of events. Furthermore, where a range of values is provided, it is understood that every intervening value, between the upper and lower limit of that range and any other stated or intervening value in that stated range is encompassed within the invention. Also, it is contemplated that any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein.
All existing subject matter mentioned herein (e.g., publications, patents, patent applications and hardware) is incorporated by reference herein in its entirety except insofar as the subject matter may conflict with that of the present invention (in which case what is present herein shall prevail).
Described herein is a modular robotic kitchen system.
Also shown in
The robotic kitchen assistant (RKA) module 30 is shown including a shielded workspace, counter-top or bin area, a robotic arm having a plurality of degrees of freedom (preferably, 6 DOF), at least one sensor or camera, and a computer operable to control the motion of the robotic arm to carry out food preparation steps as discussed further herein. Examples of an RKA and robotic arm suitable are described in U.S. patent application Ser. No. 16/490,534, filed Aug. 31, 2019, entitled “ROBOTIC KITCHEN ASSISTANT FOR PREPARING FOOD ITEMS IN A COMMERCIAL KITCHEN AND RELATED METHODS” and U.S. Patent Publication No. 20180345485, filed Aug. 10, 2018, and entitled “MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA AS PART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEM FOR RECOGNIZING AND PREPARING FOOD AND RELATED METHODS.”
The unpacking or ingredient cart 20 is shown including a shielded workspace, and four separate areas for holding ingredients or bins of ingredients. As discussed further herein, in embodiments, the ingredient cart 20 can hold multiple food items (up to 10), is robot friendly; includes face protection, and a lid or cover to close. Optionally, one or more of the separate areas are refrigerated. Additionally, in embodiments, discussed further herein, the system employs raw food packaging facilitating robot actions. By ‘food items’, it is meant to include a wide variety of types of food items whether cooked or uncooked.
The modular robotic kitchen system can operate with a wide range of cooking appliances (e.g. fryer 80, grill 90) as shown in
Preferably, in embodiments, temperature of the food items being cooked is monitored. The temperature can be input to scheduler engine, described further herein. Additionally, in embodiments, the temperature in the appliances (e.g., fryer oil, oven temperature, grill surface, etc.) can be monitored and automatically controlled, discussed further herein.
Additionally, in embodiments, the modular robotic kitchen system can include various utensils to facilitate transferring from one station or cart to another. In a particular embodiment, a fry basket is operable with the fryer and enables convenient and safe transfer of the fried items to another unit or workspace, discussed further herein.
Warming Module
Extension Module
Examples for use with embodiments of the inventions of hardware and software include, without limitation, central computer, servers, processors, memory and storage, commination interface, sensors, cameras, input devices such as keyboards or touchscreen displays, display. The processor is programmed or operable to execute various applications described herein as well as enable modules or engines for determining location and identification of food items, doneness, scheduling of steps, demand of food items, and inventory. Examples of food identification and location, scheduling, and demand modules as descried in U.S. patent application Ser. No. 16/490,534, filed Aug. 31, 2019, entitled “ROBOTIC KITCHEN ASSISTANT FOR PREPARING FOOD ITEMS IN A COMMERCIAL KITCHEN AND RELATED METHODS”, U.S. patent application Ser. No. 16/490,775, filed Sep. 3, 2019, entitled “AUGMENTED REALITY-ENHANCED FOOD PREPARATION SYSTEM AND RELATED METHODS”, and U.S. Patent Publication No. 20180345485, filed Aug. 10, 2018, and entitled “MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA AS PART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEM FOR RECOGNIZING AND PREPARING FOOD AND RELATED METHODS”, each of which is incorporated by reference in its entirety for all purposes.
Core platform additionally shows skills 140 that are enabled by the hardware and software. Collectively, the core platform is highly flexible and adaptable to perform a wide range of cooking applications 150 which may include specific cooking workflows 160 and use of specific cooking equipment 170 such as a burger workflow and use of a griddle, respectively. The core platform 110, as described further herein, is readily adaptable to run a specific cooking workflow and use the provided equipment without needing to be reworked or rewired.
In embodiments, a new cooking workflow software is downloaded to the central computer for execution. Optionally, trained models may be downloaded to the central computer or the system trains itself based on machine learning algorithms.
The monitoring system 172 is operable to continuously track the status of the system and flags anomalous behavior to be corrected by local or remote staff.
The continuous learning system 174 is operable to utilize these flagged issues to retrain the neural networks in order to improve the performance of the autonomous system for food classification.
A performance analytic system 176 is operable to aggregate at regular intervals to improve store management and give guidance on where to focus efforts. The analytics serve to determine the difference between the amount of food cooked compared to the amount of food ordered, to produce food safety and quality reports, and to report on the status of the machine and when the next maintenance cycle is due.
In embodiments, a method for packing, transporting, and unpacking raw food for preparation in kitchens includes providing custom containers designed for ergonomic access by humans and manipulation by mechanized systems.
Preferably, the raw packing system is in a centrally located distribution warehouse and is operable to quickly unpack and repack the modular carts. Additionally, the contents in each cart is tracked throughout the time the contents are in the cart using an automated tracking system.
The raw packing system can include various hardware such as a battery and power management system, a charging interface to supply power to the battery and power management system in the cart, and a wired and/or wireless communication system to maintain in-transit tracking of the cart and also to communicate with the robotic kitchen assistant modular unit, described herein.
In embodiments, an access control system is provided with the cart and operable to obtain a return merchandise authorization (RMA) and to allow the contents in the cart to be returned safely back to the distribution warehouse and repacked for a different store, without risking store-to-store contamination.
The packing and unpacking system can optionally log environmental data of the cart at all times.
The packing and unpacking system may include an environmental control system to control the temperature and other environmental conditions within he warehouse or kitchen. For example, in embodiments, the environmental control system comprises a compressor-based bidirectional heat pump, and optionally the heat pump may be a solid state heat pump using, e.g., Peltier junctions.
In embodiments, the environmental control system comprises a passive thermal reservoir utilizing ice or other similar latent heat of phase change and heavy insulation. In embodiments, a combination of the above thermal control systems are used in combination.
In embodiments, the raw food is packaged in a thermal insulative container. In particular embodiments, raw food is packaged in pillow packs that are hermetically sealed via plastic welding. The pillow pack is opened via a blade or a perforation in the packaging material, and the contents can then be dumped into cooking container, e.g. fryer basket or pot. The packaging material is then discarded.
In embodiments, the pillow pack container implementation can be grabbed with a suction cup.
In embodiments, the pillow pack container can be grabbed with a molded gripping feature designed for a custom end effector to enhance manipulability of pillow pack.
In embodiments, small reusable rigid containers are used to contain food product. Preferably, in embodiments, a freezer safe package can be opened without the use of a knife by pulling apart the bag.
With reference to
In embodiments, the freezer safe package encodes information about the product.
In embodiments, a freezer safe package is adapted to dissolve in hot oil to release contents into the oil to cook. Exemplary materials for the freezer safe bag include rice paper, starch, etc.
In embodiments, a robotic assisted method for determining the temperature of food being cooked (e.g., batch of fried foods) comprises singulating the pieces of cooked food from a batch, ranking the pieces according to size, and testing the internal temperature of the largest pieces to guarantee food safety requirements.
Step 210 states to insert a bin of the food items in a vibrating rack. With reference to
An example of a vibrating rack mechanism which allows a bin to be agitated easily is shown in
The bin shown in
Step 220 states to vibrate for 30 seconds, or until the foods items are separated from one another. Steps 210 and 220 collectively serve to singulate the food items.
Step 230 states to capture images of the food items from a plurality of cameras.
Examples of the sensors include, without limitation, cameras (optical and/or IR) and Time-of-Flight sensors (depth measurements). The array of cameras 310 serves to provide enough information to estimate volume from reconstructed 3D models, discussed further herein. Additionally, the bin can be made of highly transparent material to allow vision from the bottom.
Step 240 states to reconstruct the 3D model of the food items. The robotic temperature testing system performs this analysis using the array of cameras and performing a technique called stereo reconstruction as described in, for example, Ju Yong Chang, Haesol Park, In Kyu Park, Kyoung Mu Lee, Sang Uk Lee, GPU-friendly multi-view stereo reconstruction using surfel representation and graph cuts, Computer Vision and Image Understanding, Volume 115, Issue 5, 2011, Pages 620-634. In embodiments, the images from the plurality of cameras are fused together using Stereo Reconstruction to obtain a 3D scan of the bins and the objects therein.
In embodiments, segmentation is achieved using a neural network as in Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick, Mask R-CNN, arXiv, 2017. Analyzing the segmentation can determine whether the food items have been fully singulated as well as provide a list of objects of interest.
Step 250 states to identify the largest pieces. In embodiments, for each piece of food, the system performs a volumetric analysis. Particularly, the segmented pieces are analyzed to see which are the largest with select geometric calculations to find the largest part of a piece of food. The pieces can be ranked according thickness of the thickest part. One or more of the thickest pieces are then selected for temperature testing, discussed below.
Step 260 states to compute the optimal angle of approach and penetration depth for the temperature probe discussed further herein. This approach and penetration step is calculated based on the information, size and orientation determined from the above steps. In embodiments, it is desirable to aim towards the center of mass of the food item, and of the largest food item.
In order to test a given piece of food properly, an appropriate angle of approach and penetration depth must be selected. For homogeneous items (such as a piece of boneless chicken breast), it is sufficient to locate the largest cross-sectional area and penetrate orthogonally to the surface and up to the middle of the food item.
For items that do not have reasonably homogeneous heat capacity, such as a bone-in chicken breast, it is not sufficient to simple insert into the largest cross section area. For the example of bone-in chicken breast, it's important to these the thickest piece but avoid the bone since it heats much faster than the surrounding tissue. Therefore, a model is necessary to infer optimal angle of approach and penetration depth.
Learning a model for angle of approach and penetration depth can be accomplished either through heuristic approaches or using machine learning. With either approach, the goal is to build a model to estimate the pose and parameters of a food item. Using this model, some embodiments use heuristics to specify how to approach and penetrate.
In embodiments, a heuristic model is sometimes used, such as located the largest cross-sectional area and penetrating orthogonal to that. This type of method can work well on a variety of food items. But some food items require more complicated techniques.
Other embodiments use learning by demonstration to build a model for angle of approach and penetration. In embodiments, a thermal probe that publishes its pose in space is used by a human trainer. The human trainer goes through the motions and the pose of the thermal probe is tracked over time as the human trainer tests many pieces of a type of food item. Using these data, a model can be trained that will allow computation of the optimal angle of approach and penetration depth.
These models for computing optimal angle of approach and penetration depth are generated using shared data via the Internet. This allows multiple robotic temperature testing systems to learn more quickly.
Step 270 states to move and insert the probe. In embodiments, temperature testing is performed with a temperature testing tool 400 and probe 420 attached to the end of a robot arm 302. This robot arm 302 can have 4, 5, 6, 7, or a higher number of degrees of freedom. The robot arm can also take other configurations including but not limited to that of a SCARA arm or a delta arm.
In the embodiment shown in
The extension may be performed by various mechanisms such as, e.g., a loaded spring 440, a pneumatic actuator, or an electromagnetic actuator such as a motor. Retraction can be accomplished with a pneumatic actuator or an electromagnetic actuator. Preferably, the extension action is performed using a sufficiently fast actuator to cause the extending probe to quickly penetrate food. By moving quickly enough, the probe is able to avoid static friction altogether and operate with kinetic friction which allows for less friction overall. This mitigates undesired motion of the food item being tested that would otherwise occur during insertion of the thermal probe
The probe may be made of various materials including, e.g., stainless steel or another food-safe material with appropriate thermal properties that can be inserted into a variety of cooked foods including but not limited to bone-in chicken, chicken tenders, and chicken nuggets, hen/turkey parts, boneless chicken/turkey pieces, steaks, hamburgers, fillets, tenders, cutlets, potato wedges, etc.
In embodiments, the thermal probe has axial force sensing. This force sensing provides feedback if the probe makes contact with a bone in a piece of meat or if a probe makes contact with any other impenetrable components in a piece of food. In spring-loaded embodiments of the thermal probe, the force can be sensed by measuring the displacement of the probe from full extensions and applying Hooke's Law. In electromagnetic embodiments, current and dynamics can be measured and compared against a model of expected current.
Step 280 states to record the temperature reading.
Step 290 states to inform user testing is complete.
Additionally, in embodiments of the invention, a sanitation is performed when a piece of food is measured to be below the food-safe temperature threshold. The sanitation step may be performed variously. In one embodiment, the probe is sanitized with an attached sanitation bath. The sanitation bath uses approved chemicals to sanitize the thermal probe and flange.
The modular kitchen systems described herein may also monitor and control temperature of the appliances (e.g., a fryer or oven) during operation.
In one embodiment, a method for controlling kitchen equipment temperature includes selecting the optimal input at present time while optimizing for a time horizon based off of future thermal load prediction and oil life preservation goals.
Oil life preservation may be performed, for example, by dropping the temperature of kitchen equipment such as a fryer to extend the lifetime of consumables such as fryer oil during periods when equipment is not in use, as determined by a kitchen production forecasting system.
Additionally, the present invention includes preemptively changing thermal input into the kitchen equipment before a thermal load is applied. For example, fryer gas burner can be turned on 20 seconds before a basket of frozen fries is dropped into fryer.
Preferably, control of the equipment is automated. In embodiments, a controller utilizes a camera or sensors to track workers in the kitchen to predict when food will be added to system. The controller raises or lowers the temperature of the appliance automatically based on the location and movement of the workers.
In embodiments, the controller is connected to a production forecasting system based on various inputs. Examples of input to the production forecasting system include, without limitation: prior demand, point-of-sale data, and product stock levels.
In embodiments, the controller is connected to a robotic kitchen assistant which relays its cooking cadence over to controller for predictive temperature control.
In embodiments, the computer monitors the health of the kitchen equipment by observing effect of heat input on temperature readouts when equipment has no thermal load.
In embodiments, the robot is operable to skim contents out of the fryer to preserve the lifetime of the equipment and the oil.
In embodiments, the system determines optimum lifetime of the oil, and when the oil needs to be changed based on tracking the throughput of food cooked in the fryer.
Step 610 states to insert bin of unsorted food. With reference to
Step 620 states to place at least one packing container 730 inside packing area 704. In embodiments, one bin sits in the work area to be used for packing. Another bin sits in the work area and contains packing containers. However, the number of the bins and areas may vary.
Step 630 states to capture images of the unsorted food. The cameras or sensors 760, described herein, can be arranged above the worksurface and food items or elsewhere to aim at and obtain images from multiple angles of the unsorted food. With reference to
Indeed, in order to properly portion and plate or pack a container, a Robotic Food Packing System can see in 3D the objects inside a bin of unsorted food. This 3D imaging data can then be used to drive decisions, discussed herein, on how and what to pick out of the bin of unsorted food. High fidelity 3D vision in a Robotic Food Packing System is achieved with an array of optical cameras mounted above the Working Surface of the Cart. These cameras point at the various work areas as in
Step 640 states to reconstruct the 3D model. Preferably, as discussed above, stereo reconstruction is employed for this step.
Step 650 states to segment and classify the food items. This step may be carried out as described above.
Step 660 states to compute an optimal grasp approach for a piece of food. This step may be determined and carried out based on parameters of the end effector tool 770 and the robot arm 710, and the output from step 650.
Step 670 states to execute grasp.
Step 680 states to place food pieces in appropriate configuration in packing container 730. This step is executed by the robotic arm, and based on order information. In embodiments, pick and place is achieved using computer vision. Images are captured by video cameras and processed by convolutional neural networks. Such a network involves a series of convolutional layers and max pool layers as well as other layers. These neural networks can be trained to infer the optimal angle of approach and determine the path necessary to successfully pick up an object.
Step 690 states to remove bin of packed containers. Optionally, similar to step 610, the system is operable to interface with either a human kitchen worker or another robotic kitchen assistant to remove the bin of packed food containers from the packing area 730.
As mentioned herein, the workspace of the modular cart may be shielded to protect workers. In embodiments, and with reference to
In embodiments, various types of gripping, grasping, wedging, squeezing, clamping, scooping, ladling, skewering, and suctioning tools are used to pick up one or more pieces of food. With reference to
In embodiments, sorting and packing is performed with a gripper tool attached to the end of a robot arm. The robot arm can have 4, 5, 6, 7, or more degrees of freedom. Additionally, the robot can have other configurations including but not limited to, a SCARA or delta-type robot.
In embodiments, the robot arm may have a camera on the wrist. The data from this camera can be combined with the data from other cameras to improve the accuracy of pick and place behaviors. In embodiments, the wrist imaging sensor may be RGB, IR, or depth, or some combination of these sensors.
In embodiments, a convolutional neural network is sometimes used to identify packing containers, either in a stack or set out in preparation for packing.
In embodiments, the decision on what and how to pack is driven by external data coming in via sensors and the Internet. Packing contents are determined by recipes.
In embodiments, learning by demonstration is sometimes used to build a model for picking up food items. A human expert goes through the motions of picking up many examples of a food item or various food items. These data can be used to train a model to pick up various food objects.
In embodiments, reinforcement learning (trial and error) is used. In this process, the system makes repeated attempts at picking up food objects. The system uses these attempts to refine the grasping model and eventually the system learns to grasp a variety of objects consistently.
In embodiments, learned models for grasping are shared amongst numerous robots potentially with a wide geographic distribution.
As discussed herein, the modular robotic kitchen system includes modular carts, appliances, and transports operable to interact and communicate with one another to deliver and prepare food according to an optimal schedule and with limited waste.
With reference to
With reference to
The configuration of the hot case may vary. The hot case 950 shown in
Additionally, in embodiments, the contents of a hot case is shared with other participants in the robotic kitchen (and sometimes also with a main controller or computer) upon which scheduling decisions (e.g., scheduling the food preparation steps) are determined.
Step 1030 states demand model. Inputs 1010 to the demand model shown in
Step 1040 states schedule optimizer. An exemplary scheduling engine is described in U.S. Patent Publication No. 20180345485, entitled “MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA AS PART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEM FOR RECOGNIZING AND PREPARING FOOD AND RELATED METHODS.” In embodiments, a central controller aggregates data to drive scheduling decisions for the entire Smart Robotic Kitchen.
In embodiments, Just-in-Time production scheduling is implemented using data from all participants in the Smart Kitchen and drives mechanical devices to produce.
The scheduler then directs or instructs one or more robotic kitchen assistant 1070, 1072, 1074 to perform the various food preparation tasks as described herein.
In embodiments, the conveyor belt assembly comprises a belt, an enclosure surrounding the belt. The enclosure acts as a protective shield to protect moving parts of the conveyor from the food. Additionally, each food item is prepared on a magnetic tray. In embodiments, the conveyor belt has a series of magnets on it. The conveyor is operable to move the magnetic food tray from underneath the protective barrier through a magnetic force.
In embodiments, the conveyor system can include one or more sensors. For example, a sensor module can be arranged on one or more of the carts to obtain image data, or time of flight sensing. The sensor module optionally includes one or more CPUs and GPUs. A processor can be provided that is operable to run convolutional neural networks and geometric analysis of 3D data achieved through stereographic reconstruction, time-of-flight sensing, or other methods.
A robotic-friendly fry basket 800 for improved packing efficiency and safety, and reduced payload on humans is shown in
Some of the advantages of the basket described above includes enabling a method for containing food for cooking in a fryer while enabling computer vision localization of basket; reducing time required to clean after use; and protecting the human worker. Additionally, in embodiments, smaller baskets are provided and used with the modular robotic system. Maintaining packing efficiency in a fryer while decreasing payload requirements can be accomplished by using many smaller baskets.
It is to be understood that the modular robotic kitchen system may vary widely except as recited in the appended claims. For example, in another embodiment, and with reference to
In embodiments, the modular cart may contain a tool belt to hold a variety of tools including measuring tools, gripping tools, and calibration tools.
In embodiments, the modular cart may have several fixed fiducial markers to provide constant feedback on calibration accuracy and allow instantaneous calibration.
In embodiments, and with reference to
In the embodiment shown in
Feet 1370 are shown extending from the legs at right angles from the legs for stability. Optionally, the feet may be mounted to the floor.
The carriage and guide cooperate together to axially move the robotic arm along the guide when commanded to do so by the computer processor, which may be located locally, as described above.
Although the linear guide system shows one robotic arm, the invention is not so limited except where recited in the appended claims. The linear rail guide system may include additional robotic arms movable along the rail to further increase the effective reach of the robotic arms. The computer and sensors operate together to determine the food preparation steps, recognize and locate the food items and utensils, and to schedule and carry out the order efficiently.
Additionally, the linear guide system may be oriented variously. In embodiments, a linear guide system extends from the front towards the back (or from the top to bottom) of the cooking area. In addition to such axial motion, the robot manipulator itself enjoys several other degrees of motion (multi-axis). Consequently, the linear guide systems can perform any of the skills and applications described above such as those identified in
The linear movement may be generated using a number of different linear movement systems. In embodiments, a cleanable linear actuator design extends the reach of one or more manipulators. In one embodiment, the linear actuator is composed of a ball screw mechanism with thread and pitch size large enough to easily clean between the threads.
The frame may be made of various materials. In embodiments, the frame is formed of steel tubing, welded together.
Additionally, the linear actuator may be covered to protect it. In embodiments, a barrier is shaped to cover the sliding mechanisms from any splashes from food production. A cover allows access of the carriage to move freely along the rail.
Still other techniques may be employed by the robotic kitchen assistant to automatically remove debris from the fryer including rapidly contacting the rim of a trash receptacle with the skimmer, or brushing the skimmer with a tool.
Preferably, the robotic arm 1410 is operable to move laterally along a guide or rail mounted to the top of the frame 1440. In embodiments, this extends the reach of the robotic arm to the full length of the enclosure.
The size of the enclosed robotic kitchen system may vary. Preferably, the enclosure is small, has a small profile and footprint. In the embodiment shown in
With reference to
Indeed, it is to be understood that the shape of the bin may vary.
Consequently, when food is placed in the bin, the food tends to accumulate in the cavity and against the inner surface of lip 1448.
The bin shown in
The mounting bracket is coupled to the drive shaft 1552 of the motor 1438. For example, it can be attached by a shaft collar with a keyway. However, other means may be used to couple the mounting bracket to the drive shaft of the motor such as a press fit. When the motor is activated, the drive shaft rotates, moving the attached mounting bracket. This causes the bin to rotate from the first position to the second position or vice versa. The second position may go beyond 90° in some cases to help with certain food types which do not slide as easily. Although a motor and rotating drive shaft are described in connection with
With reference to
A sensor array 1540 is shown mounted on safety shield 1550. As described herein the sensor array may comprise a camera and supplies image data to the programmed processor for monitoring the position, contents and movement of the bins.
Step 1710 states to load bin. An example of a bin 1434 is shown in
Additionally, in embodiments, the bins are removable from the mounting brackets for convenient food scooping and cleaning. The bins may also vary in size and shape. The bins may be sized to accommodate a restaurant's menu or needs.
Bins may also be color coded. Color-coded bins add to safety by assisting food prep workers to avoid food cross-contamination (i.e., shellfish or dairy-based can remain separate from other foods). In embodiments, the bins are configured to be stackable with one another and can be stored in a stacked arrangement 1424 as shown in
Step 1720 states to classify food and determine cook schedule. As described herein, the food is classified using a programmed processor and based on image data from cameras aimed at the food bins. The processor further is operable to determine an optimal schedule to cook the food in the bin in view of the current state of the system.
Step 1730 states to position basket for catching food from bin. With reference to
Step 1740 states to automatically dump the food to the basket. With reference to
Step 1750 states to move basket to fryer and cook food. With reference to 26A, robotic arm 1410 is operable to move basket 1490 into one of the fryers 1420-1422 and the automated bins are returned to the first position shown in
The processor is operable to determine the cooking time, duration, and optionally, measure the doneness as discussed herein. The system is operable to cook multiple foods in parallel arising from the multiple automated bins and fryers. Although three automated bins are shown, more or less automated bins may be incorporated into the design. For example, a modular automated bin station may be positioned on each side of the fryers and within the safety shield.
Step 1760 states to move basket and dump food to hot holding. With reference to
Each food holding tray 2060 includes a ramp (or chute) portion 2064 which guides or channels the food dumped thereon towards the lower base portion 2062. The human workers (not shown) may access the food items in the base portion of the food stations 2060.
Although four food holding trays 2060 are shown in
Optionally, a heat lamp 2070 is arranged above one or more of the food holding trays to keep the food warm.
Optionally, a safety shield 2080 is arranged between the food bin 2010 and the human workers. The shield 2080 is shaped to cover bin, rail and motor yet permit food items to fall into the base of the food tray. More specifically, in the embodiment shown in
It is to be understood that the food item sorting systems described herein may also be operable with and/or receive food items from a wide variety of sources not limited to robotic systems. Sources may include, without limitation, human workers, automated equipment, or other systems whether robotic or non-robotic based.
Optionally, the bin may be returned to the position shown in
The sorting system desirably operates automatically to sort a wide variety of foods received from the robotic arm enclosure 2030. The sorting system includes a computer (preferably the same computer used for the robotic arm) to determine how to position and orient the food bin, and the timing to do so. In embodiments, the sorting system is operable to place food items into the different food holding trays based on the type of food.
Optionally, the system stores the location of the food, food identify, time at the location, and food status in memory. Optionally, the computer activates an alert after a threshold time or condition is met. For example, if the amount of time the food is located in the food holding tray exceeds a threshold time for freshness, an alert is activated.
The type of alert may vary. Examples of alerts include audio sounds and display graphics intended to notify the human worker to remove the food from the food holding tray. The human worker may input the status of the order or optionally, the sorting system may be provided with sensors or cameras to obtain image data of the food holding tray areas in order for the computer to compute the status and state of the food in each of the food holding trays. The state of the system is then updated for inventory in food holding areas.
Optionally, the computer is programmed to maintain inventories in the food holding trays. First, inventory states are automatically determined based on quantity observed, freshness or time in holding areas, food types, and order anticipation (e.g., increase inventory during rush periods). Preferably a trained model for the food holding trays is incorporated into the system to recognize or estimate the quantity of food present in a food holding tray. Such a model may be a computer vision model (e.g., a CNN model) and trained by placing different known quantities of food in each of the trays and instructing the model as to the actual quantity present in the tray. Features or inputs to the model include the type of food item, diameter or characteristic length, etc.
Additionally, the computer may be programmed to compute the temperature in hot food holding areas based on image data or temperature sensors. An alert or notification may be computed and sent to the human worker when the temperature exceeds a threshold value. In embodiments, the hot food holding model is trained to recognize the temperature of the food items in the food holding bins based on thermal signatures arising from thermal cameras or sensors.
Additionally, in embodiments, data is perpetually collected from the multiple sorting and automated bin stations and sent to a remote server (e.g., a cloud based server) for aggregating and/or training a comprehensive model to recognize ingredients placed in the bins, track food holding times, compute temperatures, etc.
The computer then evaluates whether to instruct or notify the worker to carry out a task. Examples of tasks include, without limitation (a) to add or supply more food to the tray based on the inventory levels or (b) to throw away the food in the food holding areas if the food is determined to be expired.
Advantages of embodiments of the robotic kitchen system with automated bins and sorting, and related methods, include without limitation: (a) robot-free safety zone, by use of an entirely enclosed system serving to protect humans from hazardous equipment and materials (no human-robot working zones); (b) contamination control, by use of color-coded bins to facilitate adherence to standard operating procedures, specifically those designed to prevent cross-contamination; (c) optimizes human labor time by automating certain steps in the cooking, namely, frying process; (d) robustness, namely, tireless low maintenance robotic action; (e) flexibility, namely, the system accepts wide range of types of foods regardless of shapes, size, and quantity; (f) safety, by physically separates human from moving robotic equipment and minimizing or eliminating need of safety sensors; (g) manually lifting baskets is eliminated as well as slip and falls in vicinity of fryers; (h) less food contact with human and equipment reduces chances of contamination; (i) automatic sorting to storage trays, and (j) modularity, the system is modular and expandable.
This claims priority to provisional application No. 63/182,912, filed May 1, 2021, entitled “AUTOMATED BIN SYSTEM FOR ACCEPTING FOOD ITEMS IN ROBOTIC KITCHEN WORKSPACE.”
| Number | Name | Date | Kind |
|---|---|---|---|
| 1760462 | Yeo | May 1930 | A |
| 2658444 | Wheeler | Nov 1953 | A |
| 3302871 | Kettering | Feb 1967 | A |
| 4015494 | Spooner et al. | Apr 1977 | A |
| 4052319 | Friedman | Oct 1977 | A |
| 4545723 | Clark | Oct 1985 | A |
| 4853771 | Witriol et al. | Aug 1989 | A |
| 4919950 | Mak | Apr 1990 | A |
| 4922435 | Cahlander et al. | May 1990 | A |
| 4979864 | Cahlander | Dec 1990 | A |
| 5132914 | Cahlander et al. | Jul 1992 | A |
| 5230279 | McFadden | Jul 1993 | A |
| 5285604 | Carlin | Feb 1994 | A |
| 5386762 | Gokey | Feb 1995 | A |
| 5389764 | Nishii et al. | Feb 1995 | A |
| 5458384 | Liu et al. | Oct 1995 | A |
| 5466025 | Mee | Nov 1995 | A |
| 5833295 | Farlow, Jr. | Nov 1998 | A |
| 5893051 | Tomohiro | Apr 1999 | A |
| D412642 | King | Aug 1999 | S |
| D492112 | Hardy | Jun 2004 | S |
| 7174830 | Dong | Feb 2007 | B1 |
| 7383963 | Svabek et al. | Jun 2008 | B2 |
| 7920962 | D'Andrea et al. | Apr 2011 | B2 |
| 7971450 | Furlanetto et al. | Jul 2011 | B2 |
| 8276505 | Buehler | Oct 2012 | B2 |
| 8610037 | Polt | Dec 2013 | B2 |
| D702084 | Matos | Apr 2014 | S |
| 8820313 | Lutes | Sep 2014 | B1 |
| 9220371 | Demirakos et al. | Dec 2015 | B1 |
| 9233470 | Bradski et al. | Jan 2016 | B1 |
| 9285589 | Osterhout et al. | Mar 2016 | B2 |
| 9483875 | Theimer et al. | Nov 2016 | B2 |
| 9538880 | Riefenstein | Jan 2017 | B2 |
| 9542621 | He et al. | Jan 2017 | B2 |
| 9785911 | Galluzzo et al. | Oct 2017 | B2 |
| 9815191 | Oleynik et al. | Nov 2017 | B2 |
| 10005184 | Gerio et al. | Jun 2018 | B2 |
| D825266 | Iorio et al. | Aug 2018 | S |
| 10112771 | D'andrea et al. | Oct 2018 | B2 |
| 10154756 | Hall et al. | Dec 2018 | B2 |
| 10293488 | Hall et al. | May 2019 | B2 |
| 10682765 | Mirkhaef et al. | Jun 2020 | B2 |
| 10919144 | Sinnet et al. | Feb 2021 | B2 |
| 11167421 | Sinnet | Nov 2021 | B2 |
| 11351673 | Zito et al. | Jun 2022 | B2 |
| 11518044 | Liu et al. | Dec 2022 | B2 |
| 11577401 | Sinnet et al. | Feb 2023 | B2 |
| 11738934 | Jain | Aug 2023 | B2 |
| 20020028127 | Hart et al. | Mar 2002 | A1 |
| 20020082924 | Koether | Jun 2002 | A1 |
| 20040011321 | Hawaj | Jan 2004 | A1 |
| 20040111321 | Kargman | Jun 2004 | A1 |
| 20040154474 | Chan | Aug 2004 | A1 |
| 20040172380 | Zhang et al. | Sep 2004 | A1 |
| 20050036668 | McLennan et al. | Feb 2005 | A1 |
| 20050049940 | Tengler et al. | Mar 2005 | A1 |
| 20050193901 | Buehler | Sep 2005 | A1 |
| 20060278216 | Gagas et al. | Dec 2006 | A1 |
| 20070122000 | Venetianer et al. | May 2007 | A1 |
| 20080110347 | Wong | May 2008 | A1 |
| 20090192921 | Hicks | Jul 2009 | A1 |
| 20090210090 | Takemitsu et al. | Aug 2009 | A1 |
| 20090262206 | Park | Oct 2009 | A1 |
| 20100132692 | Shaffer | Jun 2010 | A1 |
| 20100182136 | Pryor | Jul 2010 | A1 |
| 20100296903 | Shah et al. | Nov 2010 | A1 |
| 20110153614 | Solomon | Jun 2011 | A1 |
| 20110264266 | Kock | Oct 2011 | A1 |
| 20120024170 | Fritz-Jung et al. | Feb 2012 | A1 |
| 20130033057 | Markham | Feb 2013 | A1 |
| 20130275236 | Koke et al. | Oct 2013 | A1 |
| 20130302483 | Riefenstein | Nov 2013 | A1 |
| 20140031978 | Takata | Jan 2014 | A1 |
| 20140062112 | Cho | Mar 2014 | A1 |
| 20140089299 | Kamei et al. | Mar 2014 | A1 |
| 20140157698 | Cihak et al. | Jun 2014 | A1 |
| 20140184496 | Gribetz et al. | Jul 2014 | A1 |
| 20140203012 | Corona et al. | Jul 2014 | A1 |
| 20140234066 | Mathi et al. | Aug 2014 | A1 |
| 20140324607 | Frehn et al. | Oct 2014 | A1 |
| 20140334691 | Cho et al. | Nov 2014 | A1 |
| 20140351068 | Renfroe | Nov 2014 | A1 |
| 20140363266 | Cooper | Dec 2014 | A1 |
| 20150019354 | Chan et al. | Jan 2015 | A1 |
| 20150290795 | Oleynik | Oct 2015 | A1 |
| 20150310624 | Bulan et al. | Oct 2015 | A1 |
| 20160037958 | Freymiller et al. | Feb 2016 | A1 |
| 20160078694 | Swift | Mar 2016 | A1 |
| 20160180546 | Kim et al. | Jun 2016 | A1 |
| 20160239705 | Masood et al. | Aug 2016 | A1 |
| 20160293470 | Oremus et al. | Oct 2016 | A1 |
| 20160307459 | Chestnut et al. | Oct 2016 | A1 |
| 20160327279 | Bhogal et al. | Nov 2016 | A1 |
| 20160327281 | Bhogal et al. | Nov 2016 | A1 |
| 20160334799 | D'Andrea et al. | Nov 2016 | A1 |
| 20160368742 | Maile | Dec 2016 | A1 |
| 20170011319 | Elliot et al. | Jan 2017 | A1 |
| 20170024789 | Frehn et al. | Jan 2017 | A1 |
| 20170030515 | Johnson et al. | Feb 2017 | A1 |
| 20170116661 | Sundaram | Apr 2017 | A1 |
| 20170130968 | Nagraj et al. | May 2017 | A1 |
| 20170154803 | Wang et al. | Jun 2017 | A1 |
| 20170169315 | Vaca et al. | Jun 2017 | A1 |
| 20170178070 | Wang et al. | Jun 2017 | A1 |
| 20170206431 | Sun et al. | Jul 2017 | A1 |
| 20170252922 | Levine et al. | Sep 2017 | A1 |
| 20170290345 | Garden et al. | Oct 2017 | A1 |
| 20170305015 | Krasny et al. | Oct 2017 | A1 |
| 20170348854 | Oleynik | Dec 2017 | A1 |
| 20170364073 | Guy | Dec 2017 | A1 |
| 20180046978 | Tartal | Feb 2018 | A1 |
| 20180150661 | Hall et al. | May 2018 | A1 |
| 20180339463 | Stone et al. | Nov 2018 | A1 |
| 20180345485 | Sinnet et al. | Dec 2018 | A1 |
| 20180365630 | Seals et al. | Dec 2018 | A1 |
| 20190176338 | Zito | Jun 2019 | A1 |
| 20190200797 | Diao | Jul 2019 | A1 |
| 20190256301 | Hashimoto et al. | Aug 2019 | A1 |
| 20190297899 | Weiss | Oct 2019 | A1 |
| 20190352028 | Mirkhaef et al. | Nov 2019 | A1 |
| 20190389082 | Higo | Dec 2019 | A1 |
| 20200009638 | Asada et al. | Jan 2020 | A1 |
| 20200030966 | Hasegawa et al. | Jan 2020 | A1 |
| 20200046168 | Sinnet et al. | Feb 2020 | A1 |
| 20200047349 | Sinnet et al. | Feb 2020 | A1 |
| 20200054175 | Roy et al. | Feb 2020 | A1 |
| 20200087069 | Johnson et al. | Mar 2020 | A1 |
| 20200121125 | Zito et al. | Apr 2020 | A1 |
| 20200154948 | Grass | May 2020 | A1 |
| 20200238534 | Goldberg et al. | Jul 2020 | A1 |
| 20200254641 | Hocker et al. | Aug 2020 | A1 |
| 20200337343 | Cretors | Oct 2020 | A1 |
| 20210030199 | Olson et al. | Feb 2021 | A1 |
| 20210038025 | Almblad | Feb 2021 | A1 |
| 20210094188 | Rodionov et al. | Apr 2021 | A1 |
| 20210107724 | Cohen | Apr 2021 | A1 |
| 20210196081 | Kodali et al. | Jul 2021 | A1 |
| 20210208171 | Guarracina et al. | Jul 2021 | A1 |
| 20210276756 | Dunkel | Sep 2021 | A1 |
| 20210394371 | Ishizu et al. | Dec 2021 | A1 |
| 20220055225 | Sinnet et al. | Feb 2022 | A1 |
| 20220324119 | Kodali et al. | Oct 2022 | A1 |
| 20220346598 | Sinnet et al. | Nov 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 110279287 | Sep 2019 | CN |
| 202014001807 | Mar 2014 | DE |
| 2547286 | Aug 2017 | GB |
| 2004062750 | Feb 2004 | JP |
| 2008296308 | Dec 2008 | JP |
| 2009106734 | May 2009 | JP |
| 2009297880 | Dec 2009 | JP |
| 4655912 | Mar 2011 | JP |
| 5814305 | Nov 2015 | JP |
| 0170087 | Sep 2001 | WO |
| 2006006624 | Jan 2006 | WO |
| 2012020858 | Feb 2012 | WO |
| 2015100958 | Jul 2015 | WO |
| 2015143800 | Oct 2015 | WO |
| 2016040361 | Mar 2016 | WO |
| 2015125017 | Jun 2016 | WO |
| 2016140622 | Sep 2016 | WO |
| 2017114014 | Jul 2017 | WO |
| 2017103682 | Aug 2017 | WO |
| WO-2017218548 | Dec 2017 | WO |
| 2018031489 | Feb 2018 | WO |
| 2018165038 | Sep 2018 | WO |
| 2018165105 | Sep 2018 | WO |
| 2019079345 | Apr 2019 | WO |
| 2022256799 | Dec 2022 | WO |
| Entry |
|---|
| B. Siciliano & 0. Khatib, Handbook of Robotics, published by Springer-Verlag Berlin (2008). |
| Beucher, Serge, and Fernand Meyer. “The morphological approach to segmentation: the watershed transformation.” Optical Engineering-New Yark-Marcel Dekker Incorporated—34 (1992): 433-433. |
| Bonanni et al., “Counterintelligence: Augmented Reality Kitchen”, CHI 2005, (Apr. 2, 2005), URL: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.88.2875, (Jun. 12, 2018), XP055559956. |
| International Search Report and Written Opinion of ISA dated Jul. 11, 2018 for PCT/US2018/020948. |
| International Search Report and Written Opinion of ISA dated Jul. 20, 2018 for PCT/US2018/021066. |
| International Search Report and Written Opinion of ISA dated Aug. 17, 2022 for PCT/US2022/072666. |
| International Search Report and Written Opinion of ISA of PCT application No. PCT/US2022/071871 dated Sep. 1, 2022. |
| Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick, Mask R-CNN, arXiv, 2017. |
| Krystal B., The magic of Eatsa, explained, (Mar. 2, 2017), URL: https://www.washingtonpost.com/goingoutguide/the-magic-of-eatsa-explained-order-on-a-screen-pick-up-at-a-cubby/2017/03/02/57c95fb0-f55a-11e6-b9c9-e83fce42fb61_story.html. |
| Lucas, Bruce D., and Takeo Kanade. “An iterative image registration technique with an application to stereo vision.” (1981): 674-679. |
| Ohta, Yu-Ichi, Takeo Kanade, and Toshiyuki Sakai. “Color information for region segmentation.” Computer graphics and image processing 13.3 (1980): 222-241. |
| Paul Viola, et al., Rapid Object Detection using a Boosted Cascade of Simple Features, Accepted Conference on Computer Vision and Pattern Recognition 2001 (https://www.cs.cmu.edu/˜efros/courses/LBMV07/Papers/viola-cvpr-01.pdf). |
| Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Faster”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39 Issue 6, Jun. 2017. |
| Goodfellow et al., Generative adversarial networks, Communications of the ACM, vol. 63, Issue 11, Nov. 2020 pp. 139-144. |
| Ju Yong Chang, Haesol Park, In Kyu Park, Kyoung Mu Lee, Sang Uk Lee, GPU-friendly multi-view stereo reconstruction using surfel representation and graph cuts, Computer Vision and Image Understanding, vol. 115, Issue 5, 2011, pp. 620-634. |
| Lin, Tsung-Yi, et al. “Focal loss for dense object detection.” Proceedings of the IEEE international conference on computer vision. 2017. |
| Simon, Dan. “Kalman filtering.” Embedded systems programming 14.6 (2001): 72-79. |
| Xiao-Shan Gao et al. “Complete solution classification for the perspective-three-point problem,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, No. 8, pp. 930-943, Aug. 2003. |
| Number | Date | Country | |
|---|---|---|---|
| 20220346598 A1 | Nov 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 63182912 | May 2021 | US |
