Patent Grant
11117253
The present invention relates generally to the interdisciplinary fields of robotics and artificial intelligence, more particularly to computerized robotic food preparation systems for food preparation by digitizing the food preparation process of professional and non-professional chef dishes and subsequently replicating a chef's cooking movements, processes and techniques with real-time electronic adjustments.
Research and development in robotics have been undertaken for decades but the progress has been mostly in the heavy industrial applications like automobile manufacturing automation or military applications. Simple robotics systems have been designed for the consumer markets but have largely not seen a wide application in the home-consumer robotics space thus far. With advances in technology, combined with a population with higher incomes, the market may be ripe to create opportunities for technological advances to improve people's lives. Robotics has continued to improve automation technology with enhanced artificial intelligence and emulation of human skills and tasks in many forms.
The notion of robots replacing humans in certain areas and executing tasks humans would typically perform is an ideology in continuous evolution since robots first were developed in the 1970s. Manufacturing sectors have long used robots in teach-playback mode, where the robot is taught, via pendant or offline fixed-trajectory generation and download, which motions to copy continuously and without alteration or deviation. Companies have taken the pre-programmed trajectory-execution of computer-taught trajectories and robot motion-playback into such application domains as mixing drinks, welding or painting cars, and others. However, all of these conventional applications use a 1:1 computer-to-robot or tech-playback principle that is intended to have only the robot faithfully execute the motion-commands, which is almost always following a taught/pre-computed trajectory without deviation.
Gastronomy is an art of eating well, where a gourmet recipe blends subtly high quality ingredients and flavor appealing to all our senses. Gourmet cooking follows rules based on techniques that can be very elaborate, requiring expertise and technique, and lengthy training in some cases. In the past few years, demand for gourmet food has grown tremendously because of fast rising incomes and a generational shift in culinary awareness. However, diners still need to visit a certain restaurant or venue for gourmet dishes made by a favored chef. It would be rather advantageous to see a chef preparing your favorite dish live in action or experience a dish preparation reminiscent of a childhood dish made by your grandmother.
Accordingly, it would be desirable to have a system and method to have a chef's gourmet dish made and served conveniently to consumers in their own home(s), without the necessity to travel to each restaurant around the world to enjoy specific gourmet dishes.
Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food dish with substantially the same result as if the chef had prepared the food dish. In a first embodiment, the robotic apparatus in a standardized robotic kitchen comprises two robotic arms and hands, which replicate the precise movements of a chef in the same sequence (or substantially the same sequence) and the same timing (or substantially the same timing) to prepare a food dish based on a previously recorded software file (a recipe-script) of the chef's precise movements in preparing the same food dish. In a second embodiment, a computer-controlled cooking apparatus prepares a food dish based on a sensory-curve, such as temperature over time, which was previously recorded in a software file where the chef prepared the same food dish with the cooking apparatus with sensors for which a computer recorded the sensor values over time when the chef previously prepared the food dish on the cooking apparatus fitted with sensors. In a third embodiment, the kitchen apparatus comprises the robotic arms in the first embodiment and the cooking apparatus with sensors in the second embodiment to prepare a dish that combines both the robotic arms and one or more sensory curves, where the robotic arms are capable of quality-checking a food dish during the cooking process, for such characteristics as taste, smell, and appearance, allowing for any cooking adjustments to the preparation steps of the food dish. In a fourth embodiment, the kitchen apparatus comprises a food storage system with computer-controlled containers and container identifiers for storing and supplying ingredients for a user to prepare a food dish by following a chef's cooking instructions. In a fifth embodiment, a robotic cooking kitchen comprises a robot with arms and a kitchen apparatus in which the robot moves around the kitchen apparatus to prepare a food dish by emulating a chef's precise cooking movements, including possible real-time modifications/adaptations to the preparation process defined in the recipe-script.
A robotic cooking engine comprises detection, recording, and chef emulation cooking movements, controlling significant parameters, such as temperature and time, and processing the execution with designated appliances, equipment, and tools, thereby reproducing a gourmet dish that tastes identical to the same dish prepared by a chef and served at a specific and convenient time. In one embodiment, a robotic cooking engine provides robotic arms for replicating a chef's identical movements with the same ingredients and techniques to produce an identical tasting dish.
The underlying motivation of the present disclosure centers around humans being monitored with sensors during their natural execution of an activity and then being able to use monitoring-sensors, capturing-sensors, computers and software to generate information and commands to replicate the human activity using one or more robotic and/or automated systems. While one can conceive of multiple such activities (e.g. cooking, painting, playing an instrument, etc.), one aspect of the present disclosure is directed to the cooking of a meal; in essence a robotic meal preparation application. Monitoring the human is carried out in an instrumented application-specific setting (a standardized kitchen in this case), and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of a human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef.
The use of multimodal sensing systems is the means by which the necessary raw data is collected. Sensors capable of collecting and providing such data include environment and geometrical sensors, such as two- (cameras, etc.) and three-dimensional (lasers, sonar, etc.) sensors, as well as human motion-capture systems (human-worn camera-targets, instrumented suits/exoskeletons, instrumented gloves, etc.), as well as instrumented (sensors) and powered (actuators) equipment used during recipe creation and execution (instrumented appliances, cooking-equipment, tools, ingredient dispensers, etc.). All this data is collected by one or more distributed/central computers and processed by a variety of software processes. The algorithms will process and abstract the data to the point that a human and a computer-controlled robotic kitchen can understand the activities, tasks, actions, equipment, ingredients and methods and processes used by the human, including replication of key skills of a particular chef. The raw data is processed by one or more software abstraction engines to create a recipe-script that is both human-readable and, through further processing, machine-understandable and machine-executable, spelling out all actions and motions for all steps of a particular recipe that a robotic kitchen would have to execute. These commands range in complexity from controlling individual joints, to a particular joint-motion profile over time, to abstracted levels of commands, with lower-level motion-execution commands embedded therein, associated with specific steps in a recipe. Abstracted motion-commands (e.g. “crack an egg into the pan”, “sear to a golden color on both sides”, etc.) can be generated from the raw data, and refined and optimized through a multitude of iterative learning processes, carried out live and/or off-line, allowing the robotic kitchen systems to successfully deal with measurement-uncertainties, ingredient variations, etc., enabling complex (adaptive) mini-manipulation motions using fingered-hands mounted to robot-arms and wrists, based on fairly abstracted/high-level commands (e.g. “grab the pot by the handle”, “pour out the contents”, “grab the spoon off the countertop and stir the soup”, etc.).
The ability to create machine-executable command sequences, now contained within digital files capable of being shared/transmitted, allowing any robotic kitchen to execute them, opens up the option to execute the dish-preparation steps anywhere at any time. Hence it allows for the option to buy/sell recipes online, allowing users to access and distribute recipes on a per-use or subscription basis.
The replication of a dish prepared by a human is performed by a robotic kitchen, which is in essence a standardized replica of the instrumented kitchen used by the human chef during the creation of the dish, except that the human's actions are now carried out by a set of robotic arms and handtheed by computer-monitored and computer-controllable appliances, equipment, tools, dispensers, etc. The degree of dish-replication fidelity will thus be tightly tied to the degree to which the robotic kitchen is a replica of the kitchen (and all its elements and ingredients) in which the human chef was observed while preparing the dish.
Broadly stated, there may be provided a computer-implemented method operating on a robotic apparatus, comprising an electronic description of one or more food dishes, including the recipes for making each food dish from ingredients by a chef; for each food dish, sensing a sequence of observations of a chef's movements by a plurality of robotic sensors as the chef prepares the food dish using ingredients and kitchen equipment; detecting in the sequence of observations mini-manipulations corresponding to a sequence of movements carried out in each stage of preparing a particular food dish; transforming the sensed sequence of observations into computer readable instructions for controlling a robotic apparatus capable of performing the sequences of mini-manipulations; storing at least the sequence of instructions for mini-manipulations on electronic media for each food dish, wherein the sequence of mini-manipulations for each food dish is stored as a respective electronic record; transmitting the respective electronic record for a food dish to a robotic apparatus capable of replicating the sequence of stored mini-manipulations, corresponding to the original actions of the chef; and executing the sequence of instructions for mini-manipulations for a particular food dish by the robotic apparatus, thereby obtaining substantially the same result as the original food dish prepared by the chef, wherein executing the instructions includes sensing properties of the ingredients used in preparing the food dish.
Advantageously, the robotic apparatus in a standardized robotic kitchen has the capabilities to prepare a wide array of cuisines from around the world through a global network and database access, as compared to a chef who may specialize in one type of cuisine. The standardized robotic kitchen also is able to capture and record one of your favorite food dishes for replication by the robotic apparatus whenever you like to enjoy the food dish without the repetitive process of laboring to prepare the same dish over and over again.
The structures and methods of the present invention are disclosed in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims. These and other embodiments, features, aspects, and advantages of the invention will become better understood with regard to the following description, appended claims, and accompanying drawings.
The invention will be described with respect to specific embodiments thereof, and reference will be made to the drawings, in which:
A description of structural embodiments and methods of the present invention is provided with reference to
The following definitions apply to the elements and steps described herein. These terms may likewise be expanded upon.
Abstracted Data—refers to the abstracted recipe of utility for machine-execution which has many other data-elements that a machine needs to know for proper execution and replication. This so-called meta-data, or additional data corresponding to a particular step in the cooking process, whether it be direct sensor-data (clock-time, water-temperature, camera-image, utensil or ingredient used, etc.) or data generated through interpretation or abstraction of larger data-sets (such as a 3-dimensional range cloud from a laser used to extract the location and types of objects in the image, overlaid with texture and color maps from a camera-picture, etc.), is time-stamped and used by the robotic kitchen to set, control and monitor all processes and associated methods and equipment needed at every point in time as it steps through the sequence of steps in the recipe.
Abstracted Recipe—refers to a representation of a chef's recipe, which a human knows as represented by the use of certain ingredients, in certain sequences, prepared and combined through a sequence of processes and methods as well as skills of the human chef. An abstracted recipe used by a machine for execution in an automated way requires different types of classifications and sequences. While the overall steps carried out are identical to those of the human chef, the abstracted recipe of utility to the robotic kitchen requires that additional meta-data be a part of every step in the recipe. Such meta-data includes the cooking time, variables such as temperature (and its variations over time), oven-setting, tool/equipment used, etc. Basically a machine-executable recipe-script needs to have all possible measured variables of import to the cooking process (all measured and stored while the human chef was preparing the recipe in the chef studio) correlated to time, both overall and that within each process-step of the cooking-sequence. Hence the abstracted recipe is a representation of the cooking steps mapped into a machine-readable representation or domain, which takes the required process from the human-domain to that of the machine-understandable and machine-executable domain through a set of logical abstraction steps.
Acceleration—refers to the maximum rate of speed-change at which a robotic arm can accelerate around an axis or along a space-trajectory over a short distance.
Accuracy—refers to how closely a robot can reach a commanded position. Accuracy is determined by the difference between the absolute position of the robot compared to the commanded position. Accuracy can be improved, adjusted, or calibrated with external sensing such as sensors on a robotic hand or a real-time three-dimensional model using multiple (multi-mode) sensors.
Action Primitive—In one embodiment, the term refers to an indivisible robotic action, such as moving the robotic apparatus from location X1 to location X2, or sensing the distance from an object for food preparation without necessarily obtaining a functional outcome. In another embodiment, the term refers to an indivisible robotic action in a sequence of one or more such units for accomplishing a mini-manipulation. These are two aspects of the same definition.
Automated Dosage System—refers to dosage containers in a standardized kitchen module where a particular size of food chemical compounds (such as salt, sugar, pepper, spice, any kind of liquids, such as water, oil, essences, ketchup, etc.) that is released upon application.
Automated Storage and Delivery System—refers to storage containers in a standardized kitchen module that maintain a specific temperature and humidity for storing food; each storage container is assigned a code (e.g., a bar code) for the robotic kitchen to identify and retrieval where a particular storage container delivers the food contents stored therein.
Data Cloud—refers to a collection of sensor or data-based numerical measurement values from a particular space (three-dimensional laser/acoustic range measurement, RGB-values from a camera image, etc.) collected at certain intervals and aggregated based on a multitude of relationships, such as time, location, etc.
Degree of Freedom (“DOF”)—refers to a defined mode and/or direction in which a mechanical device or system can move. The number of degrees of freedom is equal to the total number of independent displacements or aspects of motion. The total number of degrees of freedom is doubled for two robotic arms.
Edge Detection—refers to a software-based computer program(s) capable of identifying the edges of multiple objects that may be overlapping in a two-dimensional-image of a camera yet successfully identifying their boundaries to aid in object identification and planning for grasping and handling.
Equilibrium Value—refers to the target position of a robotic appendage, such as a robotic arm where the forces acting upon it are in equilibrium, i.e. there is no net force and thus no net movement.
Execution Sequence Planner—refers to a software-based computer program(s) capable of creating a sequence of execution scripts or commands for one or more elements or systems capable of being computer controlled, such as arm(s), dispensers, appliances, etc.
Food Execution Fidelity—refers to a robotic kitchen which is intended to replicate the recipe-script generated in the chef studio by watching and measuring and understanding the steps and variables and methods and processes of the human chef, thereby trying to emulate his/her techniques and skills. The fidelity of how close the execution of the dish-preparation comes to that of the human-chef is measured by how close the robotically-prepared dish resembles the human-prepared dish as measured by a variety of subjective elements, such as consistency, color, taste, etc. The notion is that, the more closely the dish prepared by the robotic kitchen is to that prepared by the human chef, the higher the fidelity of the replication process.
Food Preparation Stage (also referred to as “Cooking stage”)—refers to a combination, either sequential or in parallel, of one or more mini-manipulations including action primitives, and computer instructions for controlling the various kitchen equipment and appliances in the standardized kitchen module; one or more food preparation stages collectively represent the entire food preparation process for a particular recipe.
Geometric Reasoning—refers to a software-based computer program(s) capable of using two-dimensional (2D)/three-dimensional (3D) surface- and/or volumetric data to reason as to the actual shape and size of a particular volume; the ability to determine or utilize boundary information also allows for inferences as to the start end of a particular geometric element and the number present (in an image or model).
Grasp Reasoning—refers to a software-based computer program(s) capable of relying on geometric and physical reasoning to plan a multi-contact (point/area/volume) contact-interaction between a robotic end-effector (gripper, link, etc.), or even tools/utensils held by the end-effector, so as to successfully and stably contact, grasp and hold the object in order to manipulate it in three-dimensional space.
Hardware Automation Device—Fixed process device capable of executing pre-programmed steps in succession without the ability to modify any of them; such devices are used for repetitive motions that are not in need of any modulation.
Ingredient management and manipulation—refers to defining each ingredient in detail (including size, shape, weight, dimensions, characteristics and properties), one or more real-time adjustments in the variables associated with the particular ingredient that may differ from the previous stored ingredient details (such as the size of a fish fillet, the dimensions of an egg, etc.), and the process in executing the different stages for the manipulation movements to an ingredient.
Kitchen Module (or Kitchen Volume)—a standardized full kitchen module with standardized sets of kitchen equipment, standardized sets of kitchen tools, standardized sets of kitchen handles, and standardized sets of kitchen containers, with predefined space and dimensions for storing, accessing, and operating each kitchen element in the standardized full kitchen module. One objective of a kitchen module is to predefine as much of the kitchen equipment, tools, handles, containers, etc. as possible so as to provide a relatively fixed kitchen platform for the movements of robotic arms and hands. Both a chef in the chef kitchen studio and a person at home with a robotic kitchen (or a person at a restaurant) uses the standardized kitchen module so as to maximize the predictability of the kitchen hardware, while minimizing the risks of differentiations, variations and deviations between the chef kitchen studio and a home robotic kitchen. Different embodiments of the kitchen module are possible, including a standalone kitchen module and an integrated kitchen module. The integrated kitchen module is fitted into a conventional kitchen area of a typical house. The kitchen module operates in at least two modes, a robotic mode and a normal (manual) mode.
Machine Learning—refers to the technology wherein a software component or program improves its performance based on experience and feedback. One kind of machine learning is reinforcement learning, often used in robotics, where desirable actions are rewarded and undesirable ones are penalized. Another kind is case-based learning, where previous solutions, e.g. sequences of actions by a human teacher or by the robot itself are remembered, together with any constraints or reasons for the solutions, and then are applied or reused in new settings. There are also additional kinds of machine learning, such as inductive and transductive methods.
Mini-Manipulation—refers to a combination (or a sequence) of one or more steps that accomplish a basic functional outcome with a threshold value of the highest level of probability (examples of threshold value as within 0.1, 0.001, or 0.001 of the optimal value). Each step can be an action primitive or another (smaller) mini-manipulation, similar to a computer program comprised of basic coding steps and other computer programs that may stand alone or serve as sub-routines. For instance, a mini-manipulation can be grasping an egg, comprised of the motor actions required for reaching out a robotic arm moving the robotic fingers into the right configuration, and applying the correct delicate amount of force for grasping—all primitive actions. Another mini-manipulation can be breaking-an-egg-with-a-knife, including the grasping mini-manipulation, followed with one robotic hand, followed by grasping-a-knife mini-manipulation with the other hand, followed by the primitive action of striking the egg with the knife using a predetermined force.
Model Elements and Classification—refers to one or more software-based computer program(s) capable of understanding elements in a scene as being items that are used or needed in different parts of a task; such as a bowl for mixing and the need for a spoon to stir, etc. Multiple elements in a scene or a world-model may be classified into groupings allowing for faster planning and task-execution.
Motion Primitives—refers to motion actions that define different levels/domains of detailed action steps, e.g. a high level motion primitive would be to grab a cup, and a low level motion primitive would be to rotate a wrist by five degrees.
Multimodal Sensing Unit—refers to a sensing unit comprised of multiple sensors capable of sensing and detection in multiple modes or electromagnetic bands or spectra, particularly capable of capturing three-dimensional position and/or motion information; the electromagnetic spectrum can range from low to high frequencies and need not be limited to that perceivable by a human being. Additional modes might include, but are not limited to, other physical senses such as touch, smell, etc.
Number of Axes—three axes are required to reach any point in space. To fully control the orientation of the end of the arm (i.e. the wrist), three additional rotational axes (yaw, pitch, and roll) are required.
Parameters—refers to variables that can take numerical values or ranges of numerical values. Three kinds of parameters are particularly relevant: parameters in the instructions to a robotic device (e.g. the force or distance in an arm movement), user settable parameters (e.g. prefers meat well done vs. medium), and chef-defined parameters (e.g. set oven temperature to 350 F).
Parameter adjustment—refers to the process of changing the values of parameters based on inputs. For instance changes in the parameters of instructions to the robotic device can be based on the properties (e.g. size, shape, orientation) of, but not limited to, the ingredients, position/orientation of kitchen tools, equipment, appliances, speed, and time duration of a mini-manipulation.
Payload or carrying capacity—refers to how much weight a robotic arm can carry and hold (or even accelerate) against the force of gravity, as a function of its endpoint location.
Physical Reasoning—refers to a software-based computer program(s) capable of relying on geometrically-reasoned data and using physical information (density, texture, typical geometry and shape) to assist an inference-engine (program) to better model the object and also predict its behavior in the real world, particularly when grasped and/or manipulated/handled.
Raw Data—refers to all measured and inferred sensory-data and representation information that is collected as part of the chef-studio recipe-generation process while watching/monitoring a human chef preparing a dish. Raw data can range from a simple data-point such as clock-time, to oven temperature (over time), camera-imagery, three-dimensional laser-generated scene representation data, to appliances/equipment used, tools employed, ingredients (type and amount) dispensed and when, etc. All the information the studio-kitchen collects from its built-in sensors and stores in raw, time-stamped form is considered raw data. Raw data is then used by other software processes to generate an even higher level of understanding and recipe-process understanding, turning raw data into additional time-stamped processed/interpreted data.
Robotic Apparatus—refers the set of robotic sensors and effectors. The effectors comprise one or more robotic arms, and one or more robotic hands for operation in the standardized robotic kitchen. The sensors comprise cameras, range sensors, force sensors (haptic sensors) that transmit their information to the processor or set of processors that control the effectors.
Recipe Cooking Process—refers to a robotic script containing abstract and detailed levels of instructions to a collection of programmable and hard automation devices, so as to allow computer-controllable devices to execute a sequenced operation within its environment (e.g. a kitchen replete with ingredients, tools, utensils and appliances).
Recipe Script—refers to a recipe script as a sequence in time containing a structure and a list of commands and execution primitives (simple to complex command software) that, when executed by the robotic kitchen elements (robot-arm, automated equipment, appliances, tools, etc.) in a given sequence, should result in the proper replication and creation of the same dish as prepared by the human chef in the studio-kitchen. Such a script is sequential in time and equivalent to the sequence employed by the human chef to create the dish, albeit in a representation that is suitable and understandable by the computer-controlled elements in the robotic kitchen.
Recipe Speed Execution—refers to managing a timeline in the execution of recipe steps in preparing a food dish by replicating a chef's movements, where the recipe steps include standardized food preparation operations (e.g., standardized cookware, standardized equipment, kitchen processors, etc.), mini-manipulations, and cooking of non-standardized objects.
Repeatability—refers to an acceptable preset margin in how accurately the robotic arms/hands can repeatedly return to a programmed position. If the technical specification in a control memory requires the robotic hand to move to a certain X-Y-Z position and within +/−0.1 mm of that position, then the repeatability is measured for the robotic hands to return to within +/−0.1 mm of the taught and desired/commanded position.
Robotic Recipe Script—refers to a computer-generated sequence of machine-understandable instructions related to the proper sequence of robotically/hard-automation execution of steps to mirror the required cooking steps in a recipe to arrive at the same end-product as if cooked by a chef.
Robotic Costume—External instrumented device(s) or clothing, such as gloves, clothing with camera-trackable markers, jointed exoskeleton, etc., used in the chef studio to monitor and track the movements and activities of the chef during all aspects of the recipe cooking process(es).
Scene Modeling—refers to a software-based computer program(s) capable of viewing a scene in one or more cameras' fields of view, and being capable of detecting and identifying objects of importance to a particular task. These objects may be pre-taught and/or be part of a computer library with known physical attributes and usage-intent.
Smart Kitchen Cookware/Equipment—refers to an item of kitchen cookware (e.g., a pot or a pan) or an item of kitchen equipment (e.g., an oven, a grill, or a faucet) with one or more sensors that prepares a food dish based on one or more graphical curves (e.g., a temperature curve, a humidity curve, etc.).
Software Abstraction Food Engine—refers to a software engine that is defined as a collection of software loops or programs, acting in concert to process input data and create a certain desirable set of output data to be used by other software engines or an end-user through some form of textual or graphical output interface. An abstraction software engine is a software program(s) focused on taking a large and vast amount of input data from a known source in a particular domain (such as three-dimensional range measurements that form a data-cloud of three-dimensional measurements as seen by one or more sensors), and then processing the data to arrive at interpretations of the data in a different domain (such as detecting and recognizing a table-surface in a data-cloud based on data having the same vertical data value, etc.), in order to identify, detect and classify data-readings as pertaining to an object in three-dimensional space (such as a table-top, cooking pot, etc.). The process of abstraction is basically defined as taking a large data set from one domain and inferring structure (such as geometry) in a higher level of space (abstracting data points), and then abstracting the inferences even further and identifying objects (pots, etc.) out of the abstracted data-sets to identify real-world elements in an image, which can then be used by other software engines to make additional decisions (handling/manipulation decisions for key objects, etc.). A synonym for “software abstraction engine” in this application could be also “software interpretation engine” or even “computer-software processing and interpretation algorithm”.
Task Reasoning—refers to a software-based computer program(s) capable of analyzing a task-description and breaking it down into a sequence of multiple machine-executable (robot or hard-automation systems) steps so as to achieve a particular end result defined in the task description.
Three-dimensional World Object Modeling and Understanding—refers to a software-based computer program(s) capable of using sensory data to create a time-varying three-dimensional model of all surfaces and volumes so as to enable it to detect, identify and classify objects within the same and understand their usage and intent.
Torque vector—refers to the torsion force upon a robotic appendage including its direction and magnitude.
Volumetric Object Inference (Engine)—refers to a software-based computer program(s) capable of using geometric data and edge-information as well as other sensory data (color, shape, texture, etc.) to allow for identification of three-dimensionality of one or more objects to aid in the object identification and classification process.
The robotic food preparation software 14 includes the multimodal three-dimensional sensors 20, a capturing module 28, a calibration module 30, a conversion algorithm module 32, a replication module 34, a quality check module 36 with a three-dimensional vision system, a same result module 38, and a learning module 40. The capturing module 28 captures the movements of the chef as the chef prepares a food dish. The calibration module 30 calibrates the robotic arms 22 and robotic hands 24 before, during and after the cooking process. The conversion algorithm module 32 is configured to convert the recorded data from a chef's movements collected in the chef studio into recipe modified data (or transformed data) for use in a robotic kitchen where robotic hands replicate the food preparation of the chef's dish. The replication module 34 is configured to replicate the chef's movements in a robotic kitchen. The quality check module 36 is configured to perform quality check functions of a food dish prepared by the robotic kitchen during, prior to, or after the food preparation process. The same result module 38 is configured to determine whether the food dish prepared by a pair of robotic arms and hands in the robotic kitchen would taste the same or substantially the same as if prepared by the chef. The learning module 40 is configured to provide learning capabilities to the computer 16 that operates the robotic arms and hands.
The standardized robotic kitchen 50 is designed for detecting, recording and emulating a chef's cooking movements, controlling significant parameters such as temperature over time, and process execution at robotic kitchen stations with designated appliances, equipment and tools. The chef kitchen 44 provides a computing kitchen environment 16 with gloves with sensors or a costume with sensors for recording and capturing a chef's 50 movements in the food preparation for a specific recipe. Upon recording the movements and recipe process of the chef 49 for a particular dish into a software recipe file in memory 52, the software recipe file is transferred from the chef kitchen 44 to the robotic kitchen 48 via a communication network 46, including a wireless network and/or a wired network connected to the Internet, so that the user (optional) 60 can purchase one or more software recipe files or the user can be subscribed to the chef kitchen 44 as a member that receives new software recipe files or periodic updates of existing software recipe files. The household robotic kitchen system 48 serves as a robotic computing kitchen environment at residential homes, restaurants, and other places in which the kitchen is built for the user 60 to prepare food. The household robotic kitchen system 48 includes the robotic cooking engine 56 with one or more robotic arms and hard-automation devices for replicating the chef's cooking actions, processes and movements based on a received software recipe file from the chef studio system 44.
The chef studio 44 and the robotic kitchen 48 represent an intricately linked teach-playback system, which has multiple levels of fidelity of execution. While the chef studio 44 generates a high-fidelity process model of how to prepare a professionally cooked dish, the robotic kitchen 48 is the execution/replication engine/process for the recipe-script created through the chef working in the chef studio. Standardization of a robotic kitchen module is a means to increase performance fidelity and success/guarantee.
The varying levels of fidelity for recipe-execution depend on the correlation of sensors and equipment (besides of course the ingredients) between those in the chef studio 44 and that in the robotic kitchen 48. Fidelity can be defined as a dish tasting identical to that prepared by a human chef (indistinguishably so) at one of the (perfect replication/execution) ends of the spectrum, while at the opposite end the dish could have one or more substantial or fatal flaws with implications to quality (overcooked meat or pasta), taste (burnt elements), edibility (incorrect consistency) or even health-implications (undercooked meat such as chicken/pork with salmonella exposure, etc.).
A robotic kitchen that has identical hardware and sensors and actuation systems that can replicate the movements and processes akin to those by the chef that were recorded during the chef-studio cooking process is more likely to result in a higher fidelity outcome. The implication here is that the setups need to be identical, which has a cost and volume implication. The robotic kitchen 48 can however still be implemented using more standardized non-computer-controlled or computer-monitored elements (pots with sensors, networked appliances such as ovens, etc.), requiring more sensor-based understanding to allow for more complex execution monitoring. Since uncertainty has now increased as to key elements (correct amount of ingredients, cooking temperatures, etc.) and processes (use of stirrer/masher in case a blender is not available in a robotic home kitchen), the guarantees of having an identical outcome to that from the chef will undoubtedly be lower.
An emphasis in the present disclosure is that the notion of a chef studio 44 coupled with a robotic kitchen is a generic concept. The level of the robotic kitchen 48 is variable all the way from a home-kitchen outfitted with a set of arms and environmental sensors, all the way to an identical replica of the studio-kitchen, where a set of arms and articulated motions, tools and appliances and ingredient-supply can replicate the chef's recipe in an almost identical fashion. The only variable to contend with will be the quality-degree of the end-result or dish in terms of quality, looks, taste, edibility and health.
A potential method to mathematically describe this correlation between the recipe-outcome and the input variables in the robotic kitchen can best be described by the function below:
Frecipe-outcome=Fstudio(I,E,P,M,V)+FRobKit(Ef,I,Re,Pmf)
The above equation relates the degree to which the outcome of a robotically-prepared recipe matches that a human chef would prepare and serve (Frecipe-outcome) to the level that the recipe was properly captured and represented by the chef studio 44 (Fstudio) based on the ingredients (I) used, the equipment (E) available to execute the chef's processes (P) and methods (M) by properly capturing all the key variables (V) during the cooking process; and how the robotic kitchen is able to represent the replication/execution process of the robotic recipe script by a function (FRobKit) that is primarily driven by the use of the proper ingredients (I), the level of equipment fidelity (Ef) in the robotic kitchen compared to that in the chef studio, the level to which the recipe-script can be replicated (Re) in the robotic kitchen, and to what extent there is an ability and need to monitor and execute corrective actions to achieve the highest process monitoring fidelity (Pmf) possible.
The functions (Fstudio) and (FRobKit) can be any combination of linear or non-linear functional formulas with constants, variables and any form of algorithmic relationships. An example for such algebraic representations for both functions could be.
Fstudio=I(fct. sin(Temp))+E(fct. Cooptop1*5)+P(fct. Circle(spoon)+V(fct. 0.5*time)
Delineating that the fidelity of the preparation process is related to the temperature of the ingredient which varies over time in the refrigerator as a sinusoidal function, the speed with which an ingredient can be heated on the cooktop on specific station at a particular multiplicative rate, and related to how well a spoon can be moved in a circular path of a certain amplitude and period, and that the process needs to be carried out at no less than ½ the speed of the human chef for the fidelity of the preparation process to be maintained.
FRobKit=Ef,(Cooktop2,Size)+I(1.25*Size+Linear(Temp))+Re(Motion-Profile)+Pmf(Sensor-Suite Correspondence)
Delineating that the fidelity of the replication process in the robotic kitchen is related to the appliance type and layout for a particular cooking-area and the size of the heating-element, the size and temperature profile of the ingredient being seared and cooked (thicker steak requiring more cooking time), while also preserving the motion-profile of any stirring and bathing motions of a particular step like searing or mousse-beating, and whether the correspondence between sensors in the robotic kitchen and the chef-studio is sufficiently high to trust the monitored sensor data to be accurate and detailed enough to provide a proper monitoring fidelity of the cooking process in the robotic kitchen during all steps in a recipe.
The outcome of a recipe is not only a function of what fidelity the human chef's cooking steps/methods/process/skills were captured with by the chef studio, but also with what fidelity these can be executed by the robotic kitchen, where each of them has key elements that impact their respective subsystem performance.
The standardized hard automation dispenser(s) 82 is a device or a series of devices that is/are programmable and/or controllable via the cooking computer 16 to feed or provide pre-packaged (known) amounts or dedicated feeds of key materials for the cooking process, such as spices (salt, pepper, etc.), liquids (water, oil, etc.) or other dry materials (flour, sugar, etc.). The standardized hard automation dispensers 82 may be located at a specific station or be able to be robotically accessed and triggered to dispense according to the recipe sequence. In other embodiments, a robotic hard automation module may be combined or sequenced in series or parallel with other such modules or robotic arms or cooking utensils. In this embodiment, the standardized robotic kitchen 50 includes robotic arms 70 and robotic hands 72 and robotic hands as controlled by the robotic food preparation engine 56 in accordance with a software recipe file stored in the memory 52 for replicating a chef's precise movements in preparing a dish to produce the same tasting dish as if the chef had prepared it himself or herself. The three-dimensional vision sensors 66 provide capability to enable three-dimensional modeling of objects, providing a visual three-dimensional model of the kitchen activities, and scanning the kitchen volume to assess the dimensions and objects within the standardized robotic kitchen 50. The retractable safety glass 68 comprises a transparent material on the robotic kitchen 50, which when in an ON state extends the safety glass around the robotic kitchen to protect surrounding human beings from the movements of robotic arms 70 and hands 72, hot water and other liquids, steam, fire and other dangers influents. The robotic food preparation engine 56 is communicatively coupled to an electronic memory 52 for retrieving a software recipe file previously sent from the chef studio system 44 for which the robotic food preparation engine 56 is configured to execute processes in preparing and replicating the cooking method and processes of a chef as indicated in the software recipe file. The combination of robotic arms 70 and robotic hands 72 serves to replicate the precise movements of the chef in preparing a dish so that the resulting food dish will taste identical (or substantially identical) to the same food dish prepared by the chef. The standardized cooking equipment 74 includes an assortment of cooking appliances 46 that are incorporated as part of the robotic kitchen 50, including, but not limited to, a stove/induction/cooktop (electric cooktop, gas cooktop, induction cooktop), an oven, a grill, a cooking steamer, and a microwave oven. The standardized cookware and sensors 76 are used as embodiments for the recording of food preparation steps based on the sensors on the cookware and cooking a food dish based on the cookware with sensors, which include a pot with sensors, a pan with sensors, an oven with sensors, and a charcoal grill with sensors. The standardized cookware 78 includes frying pans, sauté pans, grill pans, multi-pots, roasters, woks, and braisers. The robotic arms 70 and the robotic hands 72 operate the standardized handles and utensils 80 in the cooking process. In one embodiment, one of the robotic hands 72 is fitted with a standardized handle, which is attached to a fork head, a knife head, and a spoon head for selection as required. The standardized hard automation dispensers 82 are incorporated into the robotic kitchen 50 to provide for expedient (via both robot arms 70 and human use) key and common/repetitive ingredients that are easily measured/dosed out or pre-packaged. The standardized containers 86 are storage locations that store food at room temperature. The standardized refrigerator containers 88 refer to, but are not limited to, a refrigerator with identified containers for storing fish, meat, vegetables, fruit, milk, and other perishable items. The containers in the standardized containers 86 or standardized storages 88 can be coded with container identifiers from which the robotic food preparation engine 56 is able to ascertain the type of food in a container based on the container identifier. The standardized containers 86 provide storage space for non-perishable food items such as salt, pepper, sugar, oil, and other spices. Standardized cookware with sensors 76 and the cookware 78 may be stored on a shelf or a cabinet for use by the robotic arms 70 for selecting a cooking tool to prepare a dish. Typically, the raw fish, the raw meat, and vegetables are pre-cut and stored in the identified standardized storages 88. The kitchen countertop 90 provides a platform for the robotic arms 70 to handle the meat or vegetables as needed, which may or may not include cutting or chopping actions. The kitchen faucet 92 provides a kitchen sink space for washing or cleaning food in preparation for a dish. When the robotic arms 70 have completed the recipe process to prepare a dish and the dish is ready for serving, the dish is placed on a serving counter 90, which further allows for the dining environment to be enhanced by adjusting the ambient setting with the robotic arms 70, such as placement of utensils, wine glasses, and a chosen wine compatible with the meal. One embodiment of the equipment in the standardized robotic kitchen module 50 is a professional series as to increase the universal appeal to prepare various types of dishes.
The standardized robotic kitchen module 50 has as one objective the standardization of the kitchen module 50 and various components with the kitchen module itself, to ensure consistency in both the chef kitchen 44 and the robotic kitchen 48 to maximize the preciseness of recipe replication while minimizing the risks of deviations from precise replication of a recipe dish between the chef kitchen 44 and the robotic kitchen 48. One main purpose of having the standardization of the kitchen module 50 is to obtain the same result of the cooking process (or the same dish) between a first food dish prepared by the chef and a subsequent replication of the same recipe process via the robotic kitchen. Conceiving a standardized platform in the standardized robotic kitchen module 50 between the chef kitchen 44 and the robotic kitchen 48 has several key considerations: same timeline, same program or mode, and quality check. The same timeline in the standardized robotic kitchen 50 where the chef prepares a food dish at the chef kitchen 44 and the replication process by the robotic hands in the robotic kitchen 48 refers to the same sequence of manipulations, the same initial and ending time of each manipulation, and the same speed of moving an object between handling operations. The same program or mode in the standardized robotic kitchen 50 refers to the use and operation of standardized equipment during each manipulation recording and execution step. The quality check refers to three-dimensional vision sensors in the standardized robotic kitchen 50 which monitor and adjust in real time each manipulation action during the food preparation process to correct any deviation and avoid a flawed result. The adoption of the standardized robotic kitchen module 50 reduces and minimizes the risks of not obtaining the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen using robotic arms and hands. Without the standardization of a robotic kitchen module and the components within the robotic kitchen module, the increased variations between the chef kitchen 44 and the robotic kitchen 48 increase the risks of not being able to obtain the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen because more elaborate and complex adjustment algorithms will be required with different kitchen modules, different kitchen equipment, different kitchenware, different kitchen tools, and different ingredients between the chef kitchen 44 and the robotic kitchen 48.
The standardized robotic kitchen module 50 includes standardization of many aspects. First, the standardized robotic kitchen module 50 includes standardized positions and orientations (in the XYZ coordinate plane) of any type of kitchenware, kitchen containers, kitchen tools and kitchen equipment (with standardized fixed holes in the kitchen module and device positions). Secondly, the standardized robotic kitchen module 50 includes a standardized cooking volume dimension and architecture. Thirdly, the standardized robotic kitchen module 50 includes standardized equipment sets, such as an oven, a stove, a dish washer, a faucet, etc. Fourth, the standardized robotic kitchen module 50 includes standardized kitchenware, standardized cooking tools, standardized cooking devices, standardized containers, and standardized food storage in a refrigerator, in terms of shape, dimension, structure, material, capabilities, etc. Fifth, in one embodiment, the standardized robotic kitchen module 50 includes a standardized universal handle for handling any kitchenware, tools, instruments, containers, and equipment, which enable a robotic hand to hold the standardized universal handle in only one correct position, while avoiding any improper grasps or incorrect orientations. Sixth, the standardized robotic kitchen module 50 includes standardized robotic arms and hands with a library of manipulations. Seventh, the standardized robotic kitchen module 50 includes a standardized kitchen processor for standardized ingredient manipulations. Eighth, the standardized robotic kitchen module 50 includes standardized three-dimensional vision devices for creating dynamic three-dimensional vision data, as well as other possible standard sensors, for recipe recording, execution tracking, and quality check functions. Ninth, the standardized robotic kitchen module 50 includes standardized types, standardized volumes, standardized sizes, and standardized weights for each ingredient during a particular recipe execution.
The input module 50 is configured to receive any type of input information such as software recipe files sent from another computing device. The calibration module 94 is configured to calibrate itself with the robotic arms 70, the robotic hands 72, and other kitchenware and equipment components within the standardized robotic kitchen module 50. The quality check module 96 is configured to determine the quality and freshness of raw meat, raw vegetables, milk-associated ingredients and other raw foods at the time that the raw food is retrieved for cooking, as well as checking the quality of raw foods when receiving the food into the standardized food storage 88. The quality check module 96 can also be configured to conduct quality testing of an object based on senses, such as the smell of the food, the color of the food, the taste of the food, and the image or appearance of the food. The chef movements recording module 98 is configured to record the sequence and the precise movements of the chef when the chef prepares a food dish. The cookware sensor data recording module 100 is configured to record sensory data from cookware equipped with sensors (such as a pan with sensors, a grill with sensors, or an oven with sensors) placed in different zones within the cookware, thereby producing one or more sensory curves. The result is the generation of a sensory curve, such as temperature curve (and/or humidity), that reflects the temperature fluctuation of cooking appliances over time for a particular dish. The memory module 102 is configured as a storage location for storing software recipe files, for either replication of chef recipe movements or other types of software recipe files including sensory data curves. The recipe abstraction module 104 is configured to use recorded sensor data to generate machine-module specific sequenced operation profiles. The chef movements replication module 106 is configured to replicate the chef's precise movements in preparing a dish based on the stored software recipe file in the memory 52. The cookware sensory replication module 108 is configured to replicate the preparation of a food dish by following the characteristics of one or more previously recorded sensory curves which was generated when the chef 49 prepared a dish by using the standardized cookware with sensors 76. The robotic cooking module 110 is configured to control and operate standardized kitchen operations, mini-manipulations, non-standardized objects, and the various kitchen tools and equipment in the standardized robotic kitchen 50. The real time adjustment module 112 is configured to provide real-time adjustments to the variables associated with a particular kitchen operation or a mini operation so as to produce a resulting process that is a precise replication of the chef movement or a precise replication of the sensory curve. The learning module 114 is configured to provide learning capabilities to the robotic cooking engine 56 to optimize the precise replication in preparing a food dish by robotic arms 70 and the robotic hands 72, as if the food dish was prepared by a chef, using a method such as case-based (robotic) learning. The mini-manipulation library database module 116 is configured to store
a first database library of mini-manipulations. The standardized kitchen operation library database module 117 is configured to store a second database library of standardized kitchenware and how to operate this standardized kitchenware. The output module 118 is configured to send output computer files or control signals external to the robotic cooking engine.
These individual software modules generate such information (but are not thereby limited to only these modules) as (i) chef-location and cooking-station ID via a location and configuration module 152, (ii) configuration of arms (via torso), (iii) tools handled and when and how, (iv) utensils used and locations on the station through the hardware and variable abstraction module 154, (v) processes executed with them and (vi) variables (temperature, lid y/n, stirring, etc.) in need of monitoring through the process module B156, (vii) temporal (start/finish, type) distribution and (viii) types of processes (stir, fold, etc.) being applied, and (ix) ingredients added (type, amount, state of prep, etc.), through the cooking sequence and process abstraction module 158.
All this information is then used to create a machine-specific (not just for the robotic-arms, but also ingredient dispensers, tools and utensils, etc.) set of recipe instructions through the stand-alone module 160, which are organized as a script of sequential/parallel overlapping tasks to be executed and monitored. This recipe-script is stored (162) alongside the entire raw data set (164) in the data storage module 166 and is made accessible to either a remote robotic cooking station through the robotic kitchen interface module 168 or a human user 170 via a graphical user interface (GUI) 172.
The robotic kitchen 48 engages in a recipe replication process 106, whose profile depends on whether the kitchen is of a standardized or non-standardized type, which is checked by a process 186.
The robotic kitchen execution is dependent on the type of kitchen available to the user. If the robotic kitchen uses the same/identical (at least functionally) equipment as used in the in the chef studio, the recipe replication process is primarily one of using the raw data and playing it back as part of the recipe-script execution process. Should the kitchen however differ from the (ideal) standardized kitchen, the execution engine(s) will have to rely on the abstracted data to generate kitchen-specific execution sequences to try to achieve a similar step-by-step result.
Since the cooking process is continually monitored by all sensor units in the robotic kitchen via a monitoring process 194, regardless of whether the known studio equipment 196 or the mixed/atypical non-chef studio equipment 198 is being used, the system is able to make modifications as needed depending on a recipe progress check 200. In one embodiment of the standardized kitchen, raw data is typically played back through an execution module 188 using chef-studio type equipment, and the only adjustments that are expected are adaptations 202 in the execution of the script (repeat a certain step, go back to a certain step, slow down the execution, etc.) as there is a one-to-one correspondence between taught and played-back data-sets. However, in the case of the non-standardized kitchen, the chances are very high that the system will have to modify and adapt the actual recipe itself and its execution via a recipe script modification module 204, to suit the available tools/appliances 192 which differ from those in the chef studio 44 or the measured deviations from the recipe script (meat cooking too slowly, hot-spots in pot burning the roux, etc.). Overall recipe-script progress is monitored using a similar process 206, which differs depending on whether chef-studio equipment 208 or mixed/atypical kitchen equipment 210 is being used.
A non-standardized kitchen is less likely to result in a close-to-human chef cooked dish, as compared to using a standardized robotic kitchen that has equipment and capabilities reflective of those used in the studio-kitchen. The ultimate subjective decision is of course that of the human (or chef) tasting, which is a quality evaluation 212, yielding to a (subjective) quality decision 214.
A data process-mapping algorithm 220 uses the simpler (typically single-unit) variables to determine where the process action is taking place (cooktop and/or oven, fridge, etc.) and assigns a usage tag to any item/appliance/equipment being used whether intermittently or continuously. It associates a cooking step (baking, grilling, ingredient-addition, etc.) to a specific time-period and tracks when, where and which and how much of what ingredient was added. This (time-stamped) information dataset is then made available for the data-melding process during the recipe-script generation process 222.
The data extraction and mapping process 224 is primarily focused on taking two-dimensional information (such as from monocular/single-lensed cameras) and extracting key information from the same. In order to extract the important and more abstracted descriptive information from each successive image, several algorithmic processes have to be applied to this dataset. Such processing steps can include (but are not limited to) edge-detection, color and texture-mapping, and then using the domain-knowledge in the image, coupled with object-matching information (type and size) extracted from the data reduction and abstraction process 226, to allow for the identification and location of the object (whether an item of equipment or ingredient, etc.), again extracted from the data reduction and abstraction process 226, allowing one to associate the state (and all associated variables describing the same) and items in an image with a particular process-step (frying, boiling, cutting, etc.). Once this data has been extracted and associated with a particular image at a particular point in time, it can be passed to the recipe-script generation process 222 to formulate the sequence and steps within a recipe.
The data-reduction and abstraction engine (set of software routines) 226 is intended to reduce the larger three-dimensional data sets and extract from them key geometric and associative information. A first step is to extract from the large three-dimensional data point-cloud only the specific workspace area of importance to the recipe at that particular point in time. Once the data-set has been trimmed, key geometric features will be identified by a process known as template matching; this allows for the identification of such items as horizontal table-tops, cylindrical pots and pans, arm and hand locations, etc. Once typical known (template) geometric entities are determined in a data-set a process of object identification and matching proceeds to differentiate all items (pot vs. pan, etc.) and associates the proper dimensionality (size of pot or pan, etc.) and orientation of the same, and places them within the three-dimensional world model being assembled by the computer. All this abstracted/extracted information is then also shared with the data-extraction and mapping engine 224, prior to all being fed to the recipe-script generation engine 222.
The recipe-script generation engine process 222 is responsible for melding (blending/combining) all the available data and sets into a structured and sequential cooking script with clear process-identifiers (prepping, blanching, frying, washing, plating, etc.) and process-specific steps within each, which can then be translated into robotic-kitchen machine-executable command-scripts that are synchronized based on process-completion and overall cooking time and cooking progress. Data melding will at least involve, but will not solely be limited to, the ability to take each (cooking) process step and populating the sequence of steps to be executed with the properly associated elements (ingredients, equipment, etc.), methods and processes to be used during the process steps, and the associated key control—(set oven/cooktop temperatures/settings) and monitoring-variables (water or meat temperature, etc.) to be maintained and checked to verify proper progress and execution. The melded data is then combined into a structured sequential cooking script that will resemble a set of minimally descriptive steps (akin to a recipe in a magazine) but with a much larger set of variables associated with each element (equipment, ingredient, process, method, variable, etc.) of the cooking process at any one point in the procedure. The final step is to take this sequential cooking script and transform it into an identically structured sequential script that is translatable by a set of machines/robot/equipment within a robotic kitchen 48. It is this script the robotic kitchen 48 uses to execute the automated recipe execution and monitoring steps.
All raw (unprocessed) and processed data as well as the associated scripts (both structure sequential cooking-sequence script and the machine-executable cooking-sequence script) are stored in the data and profile storage unit/process 228 and time-stamped. It is from this database that the user, by way of a GUI, can select and cause the robotic kitchen to execute a desired recipe through the automated execution and monitoring engine 230, which is continually monitored by its own internal automated cooking process, with necessary adaptations and modifications to the script generated by the same and implemented by the robotic-kitchen elements, in order to arrive at a completely plated and served dish.
The mini-manipulation library is a command-software repository, where motion behaviors and processes are stored based on an off-line learning process, where the arm/wrist/finger motions and sequences to successfully complete a particular abstract task (grab the knife and then slice; grab the spoon and then stir; grab the pot with one hand and then use other hand to grab spatula and get under meat and flip it inside the pan; etc.). This repository has been built up to contain the learned sequences of successful sensor-driven motion-profiles and sequenced behaviors for the hand/wrist (and sometimes also arm-position corrections), to ensure successful completions of object (appliance, equipment, tools) and ingredient manipulation tasks that are described in a more abstract language, such as “grab the knife and slice the vegetable”, “crack the egg into the bowl”, “flip the meat over in the pan”, etc. The learning process is iterative and is based on multiple trials of a chef-taught motion-profile from the chef studio, which is then executed and iteratively modified by the offline learning algorithm module, until an acceptable execution-sequence can be shown to have been achieved. The mini-manipulation library (command software repository) is intended to have been populated (a-priori and offline) with all the necessary elements to allow the robotic-kitchen system to successfully interact with all equipment (appliances, tools, etc.) and main ingredients that require processing (steps beyond just dispensing) during the cooking process. While the human chef wore gloves with embedded haptic sensors (proximity, touch, contact-location/-force) for the fingers and palm, the robotic hands are outfitted with similar sensor-types in locations to allow their data to be used to create, modify and adapt motion-profiles to successfully execute desired motion-profiles and handling-commands.
The object-manipulation portion of the robotic-kitchen cooking process (robotic recipe-script execution software module for the interactive manipulation and handling of objects in the kitchen environment) 252 is further elaborated below. Using the robotic recipe-script database 254 (which contains data in raw, abstracted cooking-sequence and machine-executable script forms), the recipe script executor module 256 steps through a specific recipe execution-step. The configuration playback module 258 selects and passes configuration commands through to the robot arm system (torso, arm, wrist and hands) controller 270, which then controls the physical system to emulate the required configuration (joint-positions/-velocities/-torques, etc.) values.
The notion of being able to faithfully carry out proper environment interaction manipulation and handling tasks is made possible through a real-time process-verification by way of (i) 3D world modeling as well as (ii) mini-manipulation. Both the verification and manipulation steps are carried out through the addition of the robot wrist and hand configuration modifier 260. This software module uses data from the 3D world configuration modeler 262, which creates a new 3D world model at every sampling step from sensory data supplied by the multimodal sensor(s) unit(s), in order to ascertain that the configuration of the robotic kitchen systems and process matches that required by the recipe script (database); if not, it enacts modifications to the commanded system-configuration values to ensure the task is completed successfully. Furthermore, the robot wrist and hand configuration modifier 260 also uses configuration-modifying input commands from the mini-manipulation motion profile executor 264. The hand/wrist (and potentially also arm) configuration modification data fed to the configuration modifier 260 are based on the mini-manipulation motion profile executor 264 knowing what the desired configuration playback should be from 258, but then modifying it based on its 3D object model library 266 and the a-priori learned (and stored) data from the configuration and sequencing library 268 (which was built based on multiple iterative learning steps for all main object handling and processing steps).
While the configuration modifier 260 continually feeds modified commanded configuration data to the robot arm system controller 270, it relies on the handling/manipulation verification software module 272 to verify not only that the operation is proceeding properly but also whether continued manipulation/handling is necessary. In the case of the latter (answer ‘N’ to the decision), the configuration modifier 260 re-requests configuration-modification (for the wrist, hands/fingers and potentially the arm and possibly even torso) updates from both the world modeler 262 and the mini-manipulation profile executor 264. The goal is simply to verify that a successful manipulation/handling step or sequence has been successfully completed. The handling/manipulation verification software module 272 carries out this check by using the knowledge of the recipe script database F2 and the 3D world configuration modeler 262 to verify the appropriate progress in the cooking step currently being commanded by the recipe script executor 256. Once progress has been deemed successful, the recipe script index increment process 274 notifies the recipe script executor 256 to proceed to the next step in the recipe-script execution.
The multimodal sensor-unit(s) 302, comprising, but not limited to, video cameras 304, IR cameras and rangefinders 306, stereo (or even trinocular) camera(s) 308 and multi-dimensional scanning lasers 310, provide multi-spectral sensory data to the main software abstraction engines 312 (after being acquired & filtered in the data acquisition and filtering module 314). The data is used in a scene understanding module 316 to carry out multiple steps such as (but not limited to) building high- and lower-resolution (laser: high-resolution; stereo-camera: lower-resolution) three-dimensional surface volumes of the scene, with superimposed visual and IR-spectrum color and texture video information, allowing edge-detection and volumetric object-detection algorithms to infer what elements are in a scene, allowing the use of shape-/color-/texture- and consistency-mapping algorithms to run on the processed data to feed processed information to the Kitchen Cooking Process Equipment Handling Module 318. In the module 318, software-based engines are used for the purpose of identifying and three-dimensionally locating the position and orientation of kitchen tools and utensils and identifying and tagging recognizable food elements (meat, carrots, sauce, liquids, etc.) so as to generate data to let the computer build and understand the complete scene at a particular point in time so as to be used for next-step planning and process monitoring. Engines required to achieve such data and information abstraction include, but are not limited to, grasp reasoning engines, geometry reasoning engines, physical reasoning engines and task reasoning engines. Output data from both engines 316 and 318 are then used to feed the scene modeler and content classifier 320, where the 3D world model is created with all the key content required for executing the robotic cooking script executor. Once the fully-populated model of the world is understood, it can be used to feed the motion and handling planner 322 (if robotic-arm grasping and handling are necessary, the same data can be used to differentiate and plan for grasping and manipulating food and kitchen items depending on the required grip and placement) to allow for planning motions and trajectories for the arm(s) and attached end-effector(s) (grippers, multi-fingered hands). A follow-on Execution Sequence planner 324 creates the proper sequencing of task-based commands for all individual robotic/automated kitchen elements, which are then used by the robotic kitchen actuation systems 326. The entire sequence above is repeated in a continuous closed loop during the robotic recipe-script execution and monitoring phase.
In some embodiments a chef performs the same food preparation operation multiple times, yielding values of the sensor reading, and parameters in the corresponding robotic instructions that vary somewhat from one time to the next. The set of sensor readings for each sensor across multiple repetitions of the preparation of the same food dish provides a distribution with a mean, standard deviation and minimum and maximum values. The corresponding variations on the robotic instructions (also called the effector parameters) across multiple executions of the same food dish by the chef also define distributions with mean, standard deviation, minimum and maximum values. These distributions may be used to determine the fidelity (or accuracy) of subsequent robotic food preparations.
In one embodiment the estimated average accuracy of a robotic food preparation operation is given by:
Where C represents the set of Chef parameters (1st through nth) and R represents the set of Robotic Apparatus parameters (correspondingly (1st through nth). The numerator in the sum represents the difference between robotic and chef parameters (i.e. the error) and the denominator normalizes for the maximal difference). The sum gives the total normalized cumulative error
and multiplying by 1/n gives the average error. The complement of the average error corresponds to the average accuracy.
Another version of the accuracy calculation weighs the parameters for importance, where each coefficient (each αi) represents the importance of the ith parameter, the normalized cumulative error is
and the estimated average accuracy is given by:
In order to operate a mechanical robotic mechanism such as the ones described in the embodiments of this invention, a skilled artisan realizes that many mechanical and control problems need to be addressed, and the literature in robotics describes methods to do just that. The establishment of static and/or dynamic stability in a robotics system is an important consideration. Especially for robotic manipulation, dynamic stability is a strongly desired property, in order to prevent accidental breakage or movements beyond those desired or programmed. Dynamic stability is illustrated in
The cited literature addresses conditions for dynamic stability that are imported by reference into the present invention to enable proper functioning of the robotic arms. These conditions include the fundamental principle for calculating torque to the joints of a robotic arm:
where T is the torque vector (T has n components, each corresponding to a degree of freedom of the robotic arm), M is the inertial matrix of the system (M is a positive semi-definite n-by-n matrix), C is a combination of centripetal and centrifugal forces, also an n-by-n matrix, G(q) is the gravity vector, and q is the position vector. And they include finding stable points and minima, e.g. via the LaGrange equation if the robotic positions (x's) can be described by twice-differentiable functions (y's).
In order for the system comprised of the robotic arms and hands/grippers to be stable, it is important that the system be properly designed and built and have an appropriate sensing and control system which operates within the boundary of acceptable performance. The reason that this is important is that one wants to achieve the best (highest speed with highest position/velocity and force/torque tracking and all under stable conditions) performance possible given the physical system and what its controller is asking it to do.
When one speaks of proper design, the notion is one of achieving proper observability and controllability of the system. Observability implies that the key variables of the system (joint/finger positions and velocities, forces and torques) are measurable by the system, which implies one needs to have the ability to sense these variables, which in turn implies the presence and use of the proper sensing devices (internal or external). Controllability implies that one (computer in this case) have the ability to shape or control the key axes of the system based on observed parameters from internal/external sensors; this usually implies an actuator or direct/indirect control over a certain parameter by way of a motor or other computer-controlled actuation system. The ability to make the system as linear in its response as possible, thereby negating the detrimental effects of nonlinearities (stiction, backlash, hysteresis, etc.), allows for control schemes like PID gain-scheduling and nonlinear controllers like sliding-mode control to guarantee system stability and performance even in the light of system-modeling uncertainties (errors in mass/inertia estimates, dimensional geometry discretization, sensor/torque discretization anomalies, etc.) which are always present in any higher-performance control system.
Furthermore, the use of a proper computing and sampling system is significant, as the system's ability to follow rapid motions with a certain maximum frequency content is clearly related to what control bandwidth (closed-loop sampling rate of the computer control system) the entire system is able to achieve and thus the frequency-response (ability to track motions of certain speeds and motion-frequency content) the system is able to exhibit.
All the above characteristics are significant when it comes to ensuring that a highly redundant system can actually carry out the complex and dexterous tasks a human chef requires for a successful recipe-script execution, in both a dynamic and a stable fashion.
Machine learning in the context of robotic manipulation of relevance to the invention can involve well known methods for parameter adjustment, such as reinforcement learning. An alternate and preferred embodiment for this invention is a different and more appropriate learning technique for repetitive complex actions such as preparing and cooking a meal with multiple steps over time, namely case-based learning. Case-based reasoning, also known as analogical reasoning, has been developed over time.
As a general overview, case-based reasoning comprises the following steps:
A. Constructing and remembering cases. A case is a sequence of actions with parameters that are successfully carried out to achieve an objective. The parameters include distances, forces, directions, positions, and other physical or electronic measures whose values are required to successfully carry out the task (e.g. a cooking operation). First,
As depicted in
A person of skill in the art will appreciate that the probability of overall success can be low even if the probability of success of individual stages is relatively high. For instance, given 10 stages and a probability of success of each stage being 90%, the probability of overall success is (0.9)10=0.28 or 28%.
A stage in preparing a food dish comprises one or more mini-manipulations, where each mini-manipulation comprises one or more robotic actions leading to a well-defined intermediate result. For instance, slicing a vegetable can be a mini-manipulation consisting of grasping the vegetable with one hand, grasping a knife with the other, and applying repeated knife movements until the vegetable is sliced. A stage in preparing a dish can comprise one or multiple slicing mini-manipulations.
The probability of success formula applies equally well at the level of stages and at the level of mini-manipulations, so long as each mini-manipulation is relatively independent of other mini-manipulations.
In one embodiment, in order to mitigate the problem of reduced certainty of success due to potential compounding errors, standardized methods for most or all of the mini-manipulations in all of the stages are recommended. Standardized operations are ones that can be pre-programmed, pre-tested, and if necessary pre-adjusted to select the sequence of operations with the highest probability of success. Hence, if the probability of standardized methods via the mini-manipulations within stages is very high, so will be the overall probability of success of preparing the food dish, due to the prior work, until all of the steps have been perfected and tested. For instance, to return to the above example, if each stage utilizes reliable standardized methods, and its success probability is 99% (instead of 90% as in the earlier example), then the overall probability of success will be (0.99)10=90.4%, assuming there are 10 stages as before. This is clearly better than 28% probability of an overall correct outcome.
In another embodiment, more than one alternative method is provided for each stage, wherein, if one alternative fails, another alternative is tried. This requires dynamic monitoring to determine the success or failure of each stage, and the ability to have an alternate plan. The probability of success for that stage is the complement of the probability of failure for all of the alternatives, which mathematically is written as:
In the above expression si is the stage and A(si) is the set of alternatives for accomplishing Si. The probability of failure for a given alternative is the complement of the probability of success for that alternative, namely 1−P(si|aj), and the probability of all the alternatives failing is the product in the above formula. Hence, the probability that not all will fail is the complement of the product. Using the method of alternatives, the overall probability of success can be estimated as the product of each stage with alternatives, namely:
With this method of alternatives, if each of the 10 stages had 4 alternatives, and the expected success of each alternative for each stage was 90%, then the overall probability of success would be (1−(1−(0.9))4)10=0.99 or 99% versus just 28% without the alternatives. The method of alternatives transforms the original problem from a chain of stages with multiple single points of failure (if any stage fails) to one without single points of failure, since all the alternatives would need to fail in order for any given stage to fail, providing more robust outcomes.
In another embodiment, both standardized stages comprising standardized mini-manipulations, and alternate means of the food dish preparation stages are combined, yielding even more robust behavior. In such a case, the corresponding probability of success can be very high, even if alternatives are only present for some of the stages or mini-manipulations.
In another embodiment only the stages with lower probability of success are provided alternatives, in case of failure, for instance stages for which there is no very reliable standardized method, or for which there is potential variability, e.g. depending on odd-shaped materials. This embodiment reduces the burden of providing alternatives to all stages.
A predefined mini-manipulation is available to achieve each functional result (e.g., the egg is cracked). Each mini-manipulation comprises of a collection of action primitives which act together to accomplish the functional result. For example, the robot may begin by moving its hand towards the egg, touching the egg to localize its position and verify its size, and executing the movements and sensing actions necessary to grasp and lift the egg into the known and predetermined configuration.
Multiple mini-manipulations may be collected into stages such as making a sauce for convenience in understanding and organizing the recipe. The end result of executing all of the mini-manipulations to complete all of the stages is that a food dish has been replicated with a consistent result each time.
The robotic hand 72 has the RGB-D sensor 500 placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The RGB-D sensor 500 provides guidance to the robotic hand 72 in moving the robotic hand 72 toward the direction of the object and to make necessary adjustments to grab an object. Second, a sonar sensor 502f and/or a tactile pressure sensor are placed near the palm of the robotic hand 72, for detecting the distance and shape, and subsequent contact, of the object. The sonar sensor 502f can also guide the robotic hand 72 to move toward the object. Additional types of sensors in the hand may include ultrasonic sensors, lasers, radio frequency identification (RFID) sensors, and other suitable sensors. In addition, the tactile pressure sensor serves as a feedback mechanism so as to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to safely lift the object. In addition, the sonar sensor 502f in the palm of the robotic hand 72 provides a tactile sensing function to grab and handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand exerts on the knife and applies to the beef can be detected by the tactile sensor when the knife finishes slicing the beef, i.e. when the knife has no resistance, or when holding an object. The pressure distributed is not only to secure the object, but also not to break it (e.g. an egg).
Furthermore, each finger on the robotic hand 72 has haptic vibration sensors 502a-e and sonar sensors 504a-e on the respective fingertips, as shown by a first haptic vibration sensor 502a and a first sonar sensor 504a on the fingertip of the thumb, a second haptic vibration sensor 502b and a second sonar sensor 504b on the fingertip of the index finger, a third haptic vibration sensor 502c and a third sonar sensor 504c on the fingertip of the middle finger, a fourth haptic vibration sensor 502d and a fourth sonar sensor 504d on the fingertip of the ring finger, and a fifth haptic vibration sensor 502e and a fifth sonar sensor 504e on the fingertip of the pinky. Each of the haptic vibration sensors 502a, 502b, 502c, 502d and 502e can simulate different surfaces and effects by varying the shape, frequency, amplitude, duration and direction of a vibration. Each of the sonar sensors 504a, 504b, 504c, 504d and 504e provides sensing capability on the distance and shape of the object, sensing capability for the temperature or moisture, as well as feedback capability. Additional sonar sensors 504g and 504h are placed on the wrist of the robotic hand 72.
Together the thenar eminence 532 and hypothenar eminence 534 support application of large forces from the robot arm to an object in the working space such that application of these forces puts minimal stress on the robot hand joints (e.g., picture of the rolling pin). Extra joints within the palm 520 themselves are available to deform the palm. The palm 520 should deform in such a way as to enable the formation of an oblique palmar gutter for tool grasping in a way similar to a chef (typical handle grasp). The palm 520 should deform in such a way as to enable cupping, for conformable grasping of convex objects such as dishes and food materials in a manner similar to the chef, as shown by a cupping posture 542 in
Joints within the palm 520 that may support these motions include the thumb carpometacarpal joint (CMC), located on the radial side of the palm near the wrist, which may have two distinct directions of motion (flexion/extension and abduction/adduction). Additional joints required to support these motions may include joints on the ulnar side of the palm near the wrist (the fourth finger F4528 and the fifth finger F5530 CMC joints), which allow flexion at an oblique angle to support cupping motion at the hypothenar eminence 534 and formation of the palmar gutter.
The robotic palm 520 may include additional/different joints as needed to replicate the palm shape observed in human cooking motions, e.g., a series of coupled flexure joints to support formation of an arch 540 between the thenar and hypothenar eminences 532 and 534 to deform the palm 520, such as when the thumb F1522 touches the pinky finger F5530, as illustrated in
When the palm is cupped, the thenar eminence 532, the hypothenar eminence 534, and the MCP pads 536 form ridges around a palmar valley that enable the palm to close around a small spherical object (e.g., 2 cm).
The shape of the deformable palm will be described using locations of feature points relative to a fixed reference frame, as shown in
Feature points are measured by calibrated cameras mounted in the workspace as the chef performs cooking tasks. Trajectories of feature points in time are used to match the chef motion with the robot motion, including matching the shape of the deformable palm. Trajectories of feature points from the chef's motion may also be used to inform robot deformable palm design, including shape of the deformable palm surface and placement and range of motion of the joints of the robot hand.
In the embodiment as depicted in
The visual pattern consists of surface markings 552 on the robot hand or on a glove worn by the chef. These surface markings may be covered by a food safe transparent glove 554, but the surface markings 552 remain visible through the glove.
When the surface markings 552 are visible in a camera image, two-dimensional feature points may be identified within that camera image by locating convex or concave corners within the visual pattern. Each such corner in a single camera image is a two-dimensional feature point.
When the same feature point is identified in multiple camera images, the three-dimensional location of this point can be determined in a coordinate frame which is fixed with respect to the standardized robotic kitchen 50. This calculation is performed based on the two-dimensional location of the point in each image and the known camera parameters (position, orientation, field of view, etc. . . . ).
A reference frame 556 fixed to the robotic hand 72 can be obtained using a reference frame visual pattern. In one embodiment, the reference frame 556 fixed to the robotic hand 72 comprises of an origin and three orthogonal coordinate axes. It is identified by locating features of the reference frame's visual pattern in multiple cameras, and using known parameters of the reference frame visual pattern and known parameters of the cameras to extract the origin and coordinate axes.
Three-dimensional shape feature points expressed in the coordinate frame of the food preparation station can be converted into the reference frame of the robot hand once the reference frame of the robot hand is observed.
The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame fixed to the hand of the robot or the chef.
As illustrated in
Shape feature point locations are determined based on sensor signals. The sensors provide an output which allows calculation of distance in a reference frame which is attached to the magnet, which furthermore is attached to the hand of the robot or the chef.
The three-dimensional location of each shape feature point is calculated based on the sensor measurements and known parameters obtained from sensor calibration. The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame, which is fixed to the hand of the robot or the chef. For additional information on common contact regions on the human hand and function in grasping, see the material from Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura. “Patterns of static prehension in normal hands.” American Journal of Occupational Therapy 34, no. 7 (1980): 437-445, which this reference is incorporated by reference herein in its entirety.
The robotic hand 72 includes a camera sensor 684, such as an RGB-D sensor, an imaging sensor or a visual sensing device, placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The imaging sensor 682f provides guidance to the robotic hand 72 in moving the robotic hand 72 towards the direction of the object and to make necessary adjustments to grab an object. In addition, a sonar sensor, such as a tactile pressure sensor, may be placed near the palm of the robotic hand 72, for detecting the distance and shape of the object. The sonar sensor 682f can also guide the robotic hand 72 to move toward the object. Each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g includes ultrasonic sensors, laser, radio frequency identification (RFID), and other suitable sensors. In addition, each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g serves as a feedback mechanism to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to grab and lift the object. In addition, the sonar sensor 682f in the palm of the robotic hand 72 provides tactile sensing function to handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand 72 exerts on the knife and applies to the beef, allows the tactile sensor to detect when the knife finishes slicing the beef, i.e., when the knife has no resistance. The distributed pressure is not only to secure the object, but also so as not to exert too much pressure so as to, for example, not to break an egg). Furthermore, each finger on the robotic hand 72 has a sensor on the finger tip, as shown by the first sensor 682a on the finger tip of the thumb, the second sensor 682b on the finger tip of the index finger, the third sensor 682c on the finger tip of the middle finger, the fourth sensor 682d on the finger tip of the ring finger, and the fifth sensor 682f on the finger tip of the pinky. Each of the sensors 682a, 682b, 682c, 682d, 682e provide sensing capability on the distance and shape of the object, sensing capability for temperature or moisture, as well as tactile feedback capability.
The RGB-D sensor 684 and the sonar sensor 682f in the palm, plus the sonar sensors 682a, 682b, 682c, 682d, 682e in the finger tip of each finger, provide a feedback mechanism to the robotic hand 72 as a means to grab a non-standardized object, or a non-standardized kitchen tool. The robotic hands 72 may adjust the pressure to a sufficient degree to grab ahold of the non-standardized object. A program library 690 that stores sample grabbing functions 692, 694, 696 according to a specific time interval for which the robotic hand 72 can draw from in performing a specific grabbing function, is illustrated in
To create the mini-manipulation that results in cracking an egg with a knife, multiple parameter combinations must be tested to identify a set of parameters that ensure the desired functional result—that the egg is cracked—is achieved. In this example, parameters are identified to determine how to grasp and hold an egg in such a way so as not to crush it. An appropriate knife is selected through testing, and suitable placements are found for the fingers and palm so that it may be held for striking. A striking motion is identified that will successfully crack an egg. An opening motion and/or force are identified that allows a cracked egg to be opened successfully.
The teaching/learning process for the robotic apparatus involves multiple and repetitive tests to identify the necessary parameters to achieve the desired final functional result.
These tests may be performed over varying scenarios. For example, the size of the egg can vary. The location at which it is to be cracked can vary. The knife may be at different locations. The mini-manipulation must be successful in all of these variable circumstances.
Once the learning process has been completed, results are stored as a collection of action primitives that together are known to accomplish the desired functional result.
As an example of the operative relationship between the creation of a mini-manipulation in
At step 892, the computer 16 tests and validates the specific successful parameter combination for X number of times (such as one hundred times). At step 894, the computer 16 computes the number of failed results during the repeated test of the specific successful parameter combination. At step 896, the computer 16 selects the next one-time successful parameter combination from the temporary library, and returns the process back to step 892 for testing the next one-time successful parameter combination X number of times. If no further one-time successful parameter combination remains, the computer 16 stores the test results of one or more sets of parameter combinations that produce a reliable (or guaranteed) result at step 898. If there are more than one reliable sets of parameter combinations, at step 899, the computer 16 determines the best or optimal set of parameter combinations and stores the optimal set of parameter combination which is associated with the specific mini-manipulation for use in the mini-manipulation library database by the robotic apparatus in the standardized robotic kitchen 50 during the food preparation stages of a recipe.
One embodiment of the flow charts in functioning as a recipe filter, an ingredient filter, an equipment filter, an account and social network access, a personal partner page, a shopping cart page, and the information on the purchased recipe, registration setting, create a recipe are illustrated in
In
An example of partnership between users of the platform is demonstrated in
The standardized robotic kitchen 50 in
Based on the proper placement of the augmented sensor system 1854 placed somewhere in the robotic kitchen, such as on a computer-controllable railing, or on the torso of a robot with arms and hands, allows for 3D-tracking and raw data generation, both during chef-monitoring for machine-specific recipe-script generation, and monitoring the progress and successful completion of the robotically-executed steps in the stages of the dish replication in the standardized robotic kitchen 50.
The standardized robotic kitchen 50 in
The standardized robotic kitchen 50 depicts another possible configuration for the use of one or more augmented sensor systems 20. The standardized robotic kitchen 50 shows a multitude of augmented sensor systems 20 placed in the corners above the kitchen work-surface along the length of the kitchen axis with the intent to effectively cover the complete visible three-dimensional workspace of the standardized robotic kitchen 50.
The proper placement of the augmented sensor system 20 in the standardized robotic kitchen 50, allows for three-dimensional sensing, using video-cameras, lasers, sonars and other two- and three-dimensional sensor systems to enable the collection of raw data to assist in the creation of processed data for real-time dynamic models of shape, location, orientation and activity for robotic arms, hands, tools, equipment and appliances, as they relate to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50.
Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and mini-manipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.). The process of recipe-script execution based on key measurable variables is an essential part of the use of the augmented (also termed multi-modal) sensor system 20 during the execution of the replicating steps for a particular dish in a standardized robotic kitchen 50.
To access the ingredients storage-and-supply unit, part of the countertop with sliding doors can be opened, where the recipe software controls the doors and moves designated containers and ingredients to the access location where the robotic arm(s) may pick up the containers, open the lid, remove the ingredients out of the containers to a designated place, reseal the lid and move the containers back into storage. The container is moved from the access location back to its default location in the storage unit, and a new/next container item is then uploaded to the access location to be picked up.
An alternative embodiment for an ingredient storage-and-supply unit 1210 is depicted in
In
The top level 1292-1 contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc.
The counter level 1292-2 not only houses the robotic arms 70, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher.
The lower level 1292-3 houses the combination convection oven and microwave 1316, the dish-washer 1318 and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware and packing materials and cutlery.
The perspective view of the robotic kitchen 50 clearly identifies one of the many possible layouts and locations for equipment at all three levels, including the top level 1292-1 (storage pantry 1304, standardized cooking tools and ware 1320, storage ripening zone 1298, chilled storage zone 1300, and frozen storage zone 1302, the counter level 1292-2 (robotic arms 70, sink 1308, chopping/cutting area 1310, charcoal grill 1312, cooking appliances 1314 and serving counter 1306) and the lower level (dish-washer 1318 and oven and microwave 1316).
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 82 is included, a cabinet volume 1320 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage pantry volume 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level not only houses monitoring sensors 1892 and control units 1894, but also includes a counter area with a sink and electronically controllable faucet 1308, another counter area 1310 with removable working surfaces for cutting/chopping on a board, etc., a charcoal-based slatted grill 1312, and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1892 providing data to one or more control units 1896, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 3=378 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment.
At the simplest level this includes a cabinet volume 1294 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1315, the dish-washer 1318, the hard automation controlled ingredient dispensers 82 (not shown), and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 2020 while also including a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices 2022 are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not shown), and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 1316 while also including a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a (smart) charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1184 providing data to one or more control units 1186, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices (not shown) are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not showed)s, and a larger cabinet volume 1309 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1307 providing data to one or more control units 376, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
In one embodiment the emotional profile can be detected via machine learning methods based on statistical classifiers where the inputs are any measured levels of pheromones, hormones, or other features such as visual or auditory cues. If the set of features is {x1, x2, x3, . . . , xn} represented as a vector and y represents the emotional state, then the general form of an emotion-detection statistical classifier is:
Where the function f is a decision tree, a neural network, a logistic regressor, or other statistical classifier described in the machine learning literature. The first term minimizes the empirical error (the error detected while training the classifier) and the second term minimizes the complexity—e.g. Occam's razor, finding the simplest function and set of parameters p for that function that yield the desired result.
Additionally, in order to determine which pheromones or other features make the most difference (add the most value) to predicting emotional state, an active-learning criterion can be added, generally expressed as:
Where L is a “loss function”, f is the same statistical classifier as in the previous equation, and y-hat is the known outcome. We measure whether the statistical classifier performs better (smaller loss function) by addition new features, and if so keep them, otherwise not.
Parameters, values and quantities that evolve over time can be assessed to create a human emotional profile by detecting the change or transformation from one moment to the next. There are identifiable qualities to an emotional expression. A robot with emotions in response to its environment could make quicker and more effective decisions, e.g. when a robot is motivated by fear or joy or desire it might make better decisions and attain the goals more effectively and efficiently.
The robotic emotion engine replicates the human hormone emotions and pheromone emotions, either individually or in combination. Hormone emotions refer to how hormones change inside of a person's body and how that affects a person's emotions. Pheromone emotions refer to pheromones that are outside a person's body, such as smell, that affect a person's emotions. A person's emotional profile can be constructed by understanding and analyzing the hormone and pheromone emotions. The robotic emotion engine attempts to understand a person's emotions such as anger and fear by using sensors to detect a person's hormone and pheromone profile.
There are nine key physiological sign parameters to be measured in order to build a person's emotional profile: (1) sets of hormones 2221, which are secreted internally and trigger various biochemical pathways that cause certain effects, e.g. adrenalin and insulin are hormones, (2) sets of pheromones 2222, which are secreted externally, and have an effect on another person in a similar way, e.g. androstenol, androstenone and androstadienone, (3) micro expression 2223, which is a brief, involuntary facial expression shown by humans according to emotions experienced, (4) the heart rate 2224 or heart beat, e.g., when a person's heart rate increases, (5) sweat 2225 (e.g., goose bumps) e.g. face blushes and palms get sweaty and in the state of being excited or nervous, (6) pupil dilation 2226 (and iris sphincter, biliary muscle), e.g. pupil dilation for a short time in response to feelings of fear, (7) reflex movement v7, which is the movement/action primarily controlled by the spinal arc, as a response to an external stimulus, e.g. jaw jerk reflex, (8) body temperature 2228 (9) pressure 2229. The analysis 2230 on how these parameters change over a certain time 2231 may reveal a person's emotional state and profile.
When a user experiences an emotion or mood swing, physiological parameters such as hormone, heart rate, sweat, pheromones can be detected and recorded with a port connecting to a person's body, above the skin and directly to the vein. The start time and end time of the mood change can be determined by the person himself or herself as the person's emotional state changes. For example, a person initiates four manual emotion cycles and creates four timelines within a week, and as determined by the person, the first one lasts 2.8 hour from the time he tags the start till the time he tags the end. The second cycle last for 2 hours, the third one last for 0.8 hours, and the fourth one last for 1.6 hours.
The computer module 1420 includes modules that include, but are not limited to, a robotic painting engine 1352 interfaced to a painting movement emulator 1422, a painting control module 1421 that acts based on visual feedback of the painting execution processes, a memory module 1380 to store painting execution program files, algorithms 1423 for learning the selection and usage of the appropriate drawing tools, as well as an extended simulation validation and calibration module 1378.
One embodiment of the art platform standardization is defined as follows. First, standardized position and orientation (xyz) of any kind of art tools (brushes, paints, canvas, etc.) in the art platform. Second, standardized operation volume dimensions and architecture in each art platform. Third, standardized art tools set in each art platform. Fourth, standardized robotic arms and hands with a library of manipulations in each art platform. Fifth, standardized three-dimensional vision devices for creating dynamic three-dimensional vision data for painting recording and execution tracking and quality check function in each art platform. Sixth, standardized type/producer/mark/of all using paints during particular painting execution. Seventh, standardized type/producer/mark/size of canvas during particular painting execution.
One main purpose to have Standardized Art Platform is to achieve the same result of the painting process (i.e., the same painting) executing by the original painter and afterward duplicated by robotic Art Platform. Several main points to emphasize in using the standardized Art Platform: (1) have the same timeline (same sequence of manipulations, same initial and ending time of each manipulation, same speed of moving object between manipulations) of Painter and automatic robotic execution; and (2) there are quality checks (3D vision, sensors) to avoid any fail result after each manipulation during the painting process. Therefore the risk of not having the same result is reduced if the painting was done at the standardized art platform. If a non-standardized art platform is used, this will increase the risk of not having the same result (i.e. not the same painting) because adjustment algorithms may be required when the painting is not executed at not the same volume, with the same art tools, with the same paint or with the same canvas in the painter studio as in the robotic art platform.
In the case the user selects the robot engine to select the title/composer in step 1482, the engine uses its own interpretation of creativity in step 1492, to offer the human user to provide input to the selection process in step 1493. Should the human decline providing input, the robotic musician engine uses settings such as manual inputs to tonality, pitch and instrumentation as well as melodic variation in step 1499, to gather the necessary input in step 1130 to generate and upload selected instrument playing execution program files in step 1501, allowing the user to select the preferred one in step 1503, after the robotic musician engine has confirmed the selection in step 1502. The choice made by the human is then stored as a personal choice in the personal profile database in step 1504. Should the human decide to provide input to the query in step 1493, the user will be able in step 1493 to provide additional emotional input to the selection process (facial expressions, photo, news article, etc.). The input from step 194 is received by the robotic musician engine in step 1495, allowing it to proceed to step 1496, where the engine carries out a sentiment analysis related to all available input data and uploads a music selection based on the mood and style appropriate to the emotional input data from the human. Upon confirmation of selection for the uploaded music selection in step 1497 by the robotic musician engine, the user may select the ‘start’ button to play the program file for the selection in step 1498.
In the case where the human wants to be intimately involved in the selection of the title/composer, the system provides a list of performers for the selected title to the human on a display in step 1483. In step 1484 the user selects the desired performer, a choice input that the system receives in step 1485. In step 1486 the robotic musician engine generates and uploads the instrument playing execution program files, and proceeds in step 1487 to compare potential limitations between a human and a robotic musician's playing performance on a particular instrument, thereby allowing it to calculate a potential performance gap. A checking step 1488 decides whether there exists a gap. Should there be a gap, the system will suggest other selections based on the user's preference profile in step 1489. Should there be no performance gap, the robotic musician engine will confirm the selection in step 1490 and allow the user to proceed to step 1491, where the user may select the ‘start’ button to play the program file for the selection.
The disk drive unit 2240 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 246) embodying any one or more of the methodologies or functions described herein. The software 246 may also reside, completely or at least partially, within the main memory 244 and/or within the processor 226 during execution thereof the computer system 224, the main memory 228, and the instruction-storing portions of processor 226 constituting machine-readable media. The software 246 may further be transmitted or received over a network 18 via the network interface device 248.
While the machine-readable medium 244 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.
In general terms, there may be considered a method of motion capture and analysis for a robotics system, comprising sensing a sequence of observations of a person's movements by a plurality of robotic sensors as the person prepares a product using working equipment; detecting in the sequence of observations mini-manipulations corresponding to a sequence of movements carried out in each stage of preparing the product; transforming the sensed sequence of observations into computer readable instructions for controlling a robotic apparatus capable of performing the sequences of mini-manipulations; storing at least the sequence of instructions for mini-manipulations to electronic media for the product. This may be repeated for multiple products. The sequence of mini-manipulations for the product is preferably stored as an electronic record. The mini-manipulations may be abstracted parts of a multi-stage process, such as cutting an object, heating an object (in an oven or on a stove with oil or water), or similar. Then, the method may further comprise: transmitting the electronic record for the product to a robotic apparatus capable of replicating the sequence of stored mini-manipulations, corresponding to the original actions of the person. Moreover, the method may further comprise executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
In another general aspect, there may be considered a method of operating a robotics apparatus, comprising providing a sequence of pre-programmed instructions for standard mini-manipulations, wherein each mini-manipulation produces at least one identifiable result in a stage of preparing a product; sensing a sequence of observations corresponding to a person's movements by a plurality of robotic sensors as the person prepares the product using equipment; detecting standard mini-manipulations in the sequence of observations, wherein a mini-manipulation corresponds to one or more observations, and the sequence of mini-manipulations corresponds to the preparation of the product; transforming the sequence of observations into robotic instructions based on software implemented methods for recognizing sequences of pre-programmed standard mini-manipulations based on the sensed sequence of person motions, the mini-manipulations each comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; storing the sequence of mini-manipulations and their corresponding robotic instructions in electronic media. Preferably, the sequence of instructions and corresponding mini-manipulations for the product are stored as an electronic record for preparing the product. This may be repeated for multiple products. The method may further include transmitting the sequence of instructions (preferably in the form of the electronic record) to a robotics apparatus capable of replicating and executing the sequence of robotic instructions. The method may further comprise executing the robotic instructions for the product by the robotics apparatus, thereby obtaining substantially the same result as the original product prepared by the human. Where the method is repeated for multiple products, the method may additionally comprise providing a library of electronic descriptions of one or more products, including the name of the product, ingredients of the product and the method (such as a recipe) for making the product from ingredients.
Another generalized aspect provides a method of operating a robotics apparatus comprising receiving an instruction set for a making a product comprising of a series of indications of mini-manipulations corresponding to original actions of a person, each indication comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; providing the instruction set to a robotic apparatus capable of replicating the sequence of mini-manipulations; executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
A further generalized method of operating a robotic apparatus may be considered in a different aspect, comprising executing a robotic instructions script for duplicating a recipe having a plurality of product preparation movements; determining if each preparation movement is identified as a standard grabbing action of a standard tool or a standard object, a standard hand-manipulation action or object, or a non-standard object; and for each preparation movement, one or more of: instructing the robotic cooking device to access a first database library if the preparation movement involves a standard grabbing action of a standard object; instructing the robotic cooking device to access a second database library if the food preparation movement involves a standard hand-manipulation action or object; and instructing the robotic cooking device to create a three-dimensional model of the non-standard object if the food preparation movement involves a non-standard object. The determining and/or instructing steps may be particularly implemented at or by a computer system. The computing system may have a processor and memory.
Another aspect may be found in a method for product preparation by robotic apparatus, comprising replicating a recipe by preparing a product (such as a food dish) via the robotic apparatus, the recipe decomposed into one or more preparation stages, each preparation stage decomposed into a sequence of mini-manipulations and active primitives, each mini-manipulation decomposed into a sequence of action primitives. Preferably, each mini manipulation has been (successfully) tested to produce an optimal result for that mini manipulation in view of any variations in positions, orientations, shapes of an applicable object, and one or more applicable ingredients.
A further method aspect may be considered in a method for recipe script generation, comprising receiving filtered raw data from sensors in the surroundings of a standardized working environment module, such as a kitchen environment; generating a sequence of script data from the filtered raw data; and transforming the sequence of script data into machine-readable and machine-executable commands for preparing a product, the machine-readable and machine-executable commands including commands for controlling a pair of robotic arms and hands to perform a function. The function may be from the group consisting of one or more cooking stages, one or more mini-manipulations, and one or more action primitives. A recipe script generation system comprising hardware and/or software features configured to operate in accordance with this method may also be considered.
In any of these aspects, the following may be considered. The preparation of the product normally uses ingredients. Executing the instructions typically includes sensing properties of the ingredients used in preparing the product. The product may be a food dish in accordance with a (food) recipe (which may be held in an electronic description) and the person may be a chef. The working equipment may comprise kitchen equipment. These methods may be used in combination with any one or more of the other features described herein. One, more than one or all of the features of the aspects may be combined, so a feature from one aspect may be combined with another aspect for example. Each aspect may be computer-implemented and there may be provided a computer program configured to perform each method when operated by a computer or processor. Each computer program may be stored on a computer-readable medium. Additionally or alternatively, the programs may be partially or fully hardware-implemented. The aspects may be combined. There may also be provided a robotics system configured to operate in accordance with the method described in respect of any of these aspects.
In another aspect, there may be provided a robotics system, comprising: a multi-modal sensing system capable of observing human motions and generating human motions data in a first instrumented environment; and a processor (which may be a computer), communicatively coupled to the multi-modal sensing system, for recording the human motions data received from the multi-modal sensing system and processing the human motions data to extract motion primitives, preferably such that the motion primitives define operations of a robotics system. The motion primitives may be mini-manipulations, as described herein (for example in the immediately preceding paragraphs) and may have a standard format. The motion primitive may define specific types of action and parameters of the type of action, for example a pulling action with a defined starting point, end point, force and grip type. Optionally, there may be further provided a robotics apparatus, communicatively coupled to the processor and/or multi-modal sensing system. The robotics apparatus may be capable of using the motion primitives and/or the human motions data to replicate the observed human motions in a second instrumented environment.
In a further aspect, there may provided a robotics system, comprising: a processor (which may be a computer), for receiving motion primitives defining operations of a robotics system, the motion primitives being based on human motions data captured from human motions; and a robotics system, communicatively coupled to the processor, capable of using the motion primitives to replicate human motions in an instrumented environment. It will be understood that these aspects may be further combined.
A further aspect may be found in a robotics system comprising: first and second robotic arms; first and second robotic hands, each hand having a wrist coupled to a respective arm, each hand having a palm and multiple articulated fingers, each articulated finger on the respective hand having at least one sensor; and first and second gloves, each glove covering the respective hand having a plurality of embedded sensors. Preferably the robotics system is a robotic kitchen system.
There may further be provided, in a different but related aspect, a motion capture system, comprising: a standardized working environment module, preferably a kitchen; plurality of multi-modal sensors having a first type of sensors configured to be physically coupled to a human and a second type of sensors configured to be spaced away from the human. One or more of the following may be the case: the first type of sensors may be for measuring the posture of human appendages and sensing motion data of the human appendages; the second type of sensors may be for determining a spatial registration of the three-dimensional configurations of one or more of the environment, objects, movements, and locations of human appendages; the second type of sensors may be configured to sense activity data; the standardized working environment may have connectors to interface with the second type of sensors; the first type of sensors and the second type of sensors measure motion data and activity data, and send both the motion data and the activity data to a computer for storage and processing for product (such as food) preparation.
An aspect may additionally or alternatively be considered in a robotic hand coated with a sensing gloves, comprising: five fingers; and a palm connected to the five fingers, the palm having internal joints and a deformable surface material in three regions; a first deformable region disposed on a radial side of the palm and near the base of the thumb; a second deformable region disposed on a ulnar side of the palm, and spaced apart from the radial side; and a third deformable region disposed on the palm and extend across the base of the fingers. Preferably, the combination of the first deformable region, the second deformable region, the third deformable region, and the internal joints collectively operate to perform a mini manipulation, particularly for food preparation.
In respect of any of the above system, device or apparatus aspects, there may further be provided method aspects comprising steps to carry out the functionality of the system. Additionally or alternatively, optional features may be found based on any one or more of the features described herein with respect to other aspects
physalis
boletus
Hibiscus
oxalis
nasturtium seeds
Nigella seeds
The present invention has been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the invention may be practiced in other embodiments. The particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. The system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements, or entirely in software elements. The particular division of functionality between the various systems components described herein is merely example and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
In various embodiments, the present invention can be implemented as a system or a method for performing the above-described techniques, either singly or in any combination. The combination of any specific features described herein is also provided, even if that combination is not explicitly described. In another embodiment, the present invention can be implemented as a computer program product comprising a computer-readable storage medium and computer program code, encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-described techniques.
As used herein, any reference to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally perceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, transformed, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that, throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “displaying” or “determining” or the like refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware, and/or hardware, and, when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers and/or other electronic devices referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
The algorithms and displays presented herein are not inherently related to any particular computer, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references above to specific languages are provided for disclosure of enablement and best mode of the present invention.
In various embodiments, the present invention can be implemented as software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device can include, for example, a processor, an input device (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination thereof), an output device (such as a screen, speaker, and/or the like), memory, long-term storage (such as magnetic storage, optical storage, and/or the like), and/or network connectivity, according to techniques that are well known in the art. Such an electronic device may be portable or non-portable. Examples of electronic devices that may be used for implementing the invention include a mobile phone, personal digital assistant, smartphone, kiosk, desktop computer, laptop computer, consumer electronic device, television, set-top box, or the like. An electronic device for implementing the present invention may use an operating system such as, for example, iOS available from Apple Inc. of Cupertino, Calif., Android available from Google Inc. of Mountain View, Calif., Microsoft Windows 7 available from Microsoft Corporation of Redmond, Wash., webOS available from Palm, Inc. of Sunnyvale, Calif., or any other operating system that is adapted for use on the device. In some embodiments, the electronic device for implementing the present invention includes functionality for communication over one or more networks, including for example a cellular telephone network, wireless network, and/or computer network such as the Internet.
Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.
As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
The terms “a” or “an,” as used herein, are defined as one or more than one. The term “plurality,” as used herein, is defined as two or more than two. The term “another,” as used herein, is defined as at least a second or more.
An ordinary artisan should require no additional explanation in developing the methods and systems described herein but may find some possibly helpful guidance in the preparation of these methods and systems by examining standardized reference works in the relevant art.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised which do not depart from the scope of the present invention as described herein. It should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. The terms used should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims but should be construed to include all methods and systems that operate under the claims set forth herein below. Accordingly, the invention is not limited by the disclosure, but instead its scope is to be determined entirely by the following claims.
This application is a divisional of U.S. patent application Ser. No. 14/627,900, filed on 20 Feb. 2015, now issued as U.S. Pat. No. 9,815,191, which claims priority to U.S. Provisional Application Ser. No. 62/116,563 entitled “Method and System for Food Preparation in a Robotic Cooking Kitchen,” filed on 16 Feb. 2015, U.S. Provisional Application Ser. No. 62/113,516 entitled “Method and System for Food Preparation in a Robotic Cooking Kitchen,” filed on 8 Feb. 2015, U.S. Provisional Application Ser. No. 62/109,051 entitled “Method and System for Food Preparation in a Robotic Cooking Kitchen,” filed on 28 Jan. 2015, U.S. Provisional Application Ser. No. 62/104,680 entitled “Method and System for Robotic Cooking Kitchen,” filed on 16 Jan. 2015, U.S. Provisional Application Ser. No. 62/090,310 entitled “Method and System for Robotic Cooking Kitchen,” filed on 10 Dec. 2014, U.S. Provisional Application Ser. No. 62/083,195 entitled “Method and System for Robotic Cooking Kitchen,” filed on 22 Nov. 2014, U.S. Provisional Application Ser. No. 62/073,846 entitled “Method and System for Robotic Cooking Kitchen,” filed on 31 Oct. 2014, U.S. Provisional Application Ser. 62/055,799 entitled “Method and System for Robotic Cooking Kitchen,” filed on 26 Sep. 2014, U.S. Provisional Application Ser. No. 62/044,677, entitled “Method and System for Robotic Cooking Kitchen,” filed on 2 Sep. 2014, U.S. Provisional Application Ser. No. 62/024,948 entitled “Method and System for Robotic Cooking Kitchen,” filed on 15 Jul. 2014, U.S. Provisional Application Ser. No. 62/013,691 entitled “Method and System for Robotic Cooking Kitchen,” filed on 18 Jun. 2014, U.S. Provisional Application Ser. No. 62/013,502 entitled “Method and System for Robotic Cooking Kitchen,” filed on 17 Jun. 2014, U.S. Provisional Application Ser. No. 62/013,190 entitled “Method and System for Robotic Cooking Kitchen,” filed on 17 Jun. 2014, U.S. Provisional Application Ser. No. 61/990,431 entitled “Method and System for Robotic Cooking Kitchen,” filed on 8 May 2014, U.S. Provisional Application Ser. No. 61/987,406 entitled “Method and System for Robotic Cooking Kitchen,” filed on 1 May 2014, U.S. Provisional Application Ser. No. 61/953,930 entitled “Method and System for Robotic Cooking Kitchen,” filed on 16 Mar. 2014, and U.S. Provisional Application Ser. No. 61/942,559 entitled “Method and System for Robotic Cooking Kitchen,” filed on 20 Feb. 2014, the disclosures of which are incorporated herein by reference in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7107123 | Watanabe et al. | Sep 2006 | B2 |
| 7174830 | Dong | Feb 2007 | B1 |
| 7673916 | Greenhill et al. | Mar 2010 | B2 |
| 8483880 | de la Rosa Tames et al. | Jul 2013 | B2 |
| 8660695 | De La Rosa Tames et al. | Feb 2014 | B2 |
| 8882783 | Oostman | Nov 2014 | B2 |
| 9131807 | Roy et al. | Sep 2015 | B2 |
| 20040172380 | Zhang | Sep 2004 | A1 |
| 20050193901 | Buehler | Sep 2005 | A1 |
| 20100010703 | Coats et al. | Jan 2010 | A1 |
| 20100076708 | Hukkeri et al. | Mar 2010 | A1 |
| 20100076709 | Hukkeri et al. | Mar 2010 | A1 |
| 20100076710 | Hukkeri et al. | Mar 2010 | A1 |
| 20110040408 | De La Rosa Tames et al. | Feb 2011 | A1 |
| 20110158804 | Smith | Jun 2011 | A1 |
| 20110311102 | Mcdaniel et al. | Dec 2011 | A1 |
| 20120017718 | Greenhill | Jan 2012 | A1 |
| 20120053831 | Bibhrajit | Mar 2012 | A1 |
| 20120130588 | Hukkeri | May 2012 | A1 |
| 20120130598 | Hukkeri et al. | May 2012 | A1 |
| 20120299695 | McDaniel | Nov 2012 | A1 |
| 20130006482 | Hukkeri et al. | Jan 2013 | A1 |
| 20130145911 | Rother | Jun 2013 | A1 |
| 20130173201 | Hukkeri et al. | Jul 2013 | A1 |
| 20130313844 | De La Rosa Tames et al. | Nov 2013 | A1 |
| 20130345873 | Blumberg et al. | Dec 2013 | A1 |
| 20140154384 | Vardakostas et al. | Jun 2014 | A1 |
| 20140170275 | Bordin | Jun 2014 | A1 |
| 20140324607 | Frehn et al. | Oct 2014 | A1 |
| 20150053097 | Vardakostas et al. | Feb 2015 | A1 |
| 20150075391 | Vardakostas et al. | Mar 2015 | A1 |
| 20150114236 | Roy | Apr 2015 | A1 |
| 20150164131 | Vardakostas et al. | Jun 2015 | A1 |
| 20150183535 | Vardakostas et al. | Jul 2015 | A1 |
| Number | Date | Country |
|---|---|---|
| 2003074238 | Sep 2003 | WO |
| 2003080297 | Oct 2003 | WO |
| 2006037951 | Apr 2006 | WO |
| 2009030922 | Mar 2009 | WO |
| 2009030922 | Jun 2009 | WO |
| 2010092321 | Aug 2010 | WO |
| 2010092321 | Nov 2010 | WO |
| Entry |
|---|
| Bates, Claire, “How do you like your pancakes? Robot cooks breakfast at exhibit,” DailyMail.com, Nov. 27, 2008, Daily Mail Online, Retrieved on Sep. 29, 2015; 18 pages. |
| Bates, Claire, “Don't fancy the washing up? This robot could do the dishes for you,” DailyMail.com, Dec. 19, 2008, Daily Mail Online, Retrieved on Sep. 29, 2015; 19 pages. |
| Daily Mail Reporter, “Flipping brilliant! Now Japan has invented a robot that can cook you pancakes for breakfast,” DailyMail.com, Jun. 9, 2009, Daily Mail Online, Retrieved on Sep. 29, 2015; 21 pages. |
| Abry, Vincent, “Putting robotics at your service,” Cooking and Domestic Robot—Robotshop Blog,Feb. 14, 2010, Retrieved on Sep. 29, 2015; 12 pages. |
| Waibel, Markus, “Robots Make Bavarian Breakfast Together,” IEEE Spectrum, Jun. 11, 2011, Retrieved on Sep. 29, 2015; 3 pages. |
| Kerns, Josh, “Why robots could soon replace fast food workers demanding higher minimum wage,” Mynorthwest.com Online Blog, Jul. 11, 2013, Retrieved on Sep. 29, 2015; 3 pages. |
| McNicoll, Arion, “Meet the robot chef who ‘prints’ cookies,” CNN.com, Sep. 2, 2013, Retrieved on Sep. 29, 2015; 4 pages. |
| Aving Global Network, “Siemens to present its liftMatic wall-mounted oven AVING USA,” AVINGGlobalNews.network. Retrieved on Mar. 3, 2015, http://us.aving.net/news/view.php?articleId=135357; 6 pages. |
| Cha, Elizabeth et al., “Robots in the Home: Qualitative and Quantitative Insights into Kitchen Organization,” HRI' 15, Mar. 2-5, 2015, Portland, Oregon, USA; 8 pages. |
| The Shadow Robot Company, “Shadow Dexterous Hand Technical Specification,” Shadow Dexterous Hand E1 Series (E1M3R, E1M3L, E1P1R, E1P1L), Released on Jan. 1, 2013; 14 pages. |
| Treehugger, “It Slices! It Dices! It Cleans UP After Itself! Introducing Panasonic's Robot Kitchen,” Retrieved on Mar. 3, 2015, http://www.treehugger.com/kitchen-design/it-slices-it-dices-it-cleans-up-after-itself-introducing-panasonics-robot-kitchen.html; 4 pages. |
| Yang, Yezhou et al. “Robot Learning Manipulation Action Plans by “Watching” Unconstrained Videos from the World Wide Web,” Association for the Advancement of Artificial Intelligence (www.aaai.org), 2005; 7 pages. |
| Mark Oleynik, International Search Report and Written Opinion of the International Searching Authority corresponding to PCT/EP2015/001704, The European Patent Office, P.B. 5818 Patentlaan 2, dated Feb. 19, 2016; 13 pages. |
| Mark Oleynik, International Search Report corresponding to PCT/IB2015/000379, The European Patent Office, P.B. 5818 Patentlaan 2, dated Apr. 13, 2016; 6 pages. |
| MBL Limited, International Search Report on Patentability (Chapter I of the Patent Cooperation Treaty) and Written Opinion of the International Searching Authority corresponding to PCT/EP2015/001704, The International Bureau of WIPO, 34, chemin des Colombettes, 1211 Geneva 20, Switzerland, dated Mar. 16, 2017; 10 pages. |
| Mark Oleynik, English Translation of the International Preliminary Report on Patentability and Written Opinion of the International Searching Authority corresponding to PCT/IB2015/000379, The International Bureau of WIPO, dated Sep. 1, 2016; 18 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20180257219 A1 | Sep 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62116563 | Feb 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14627900 | Feb 2015 | US |
| Child | 15782230 | US |
