Patent Application
20210127943
The present disclosure relates to the field of objects placement recommendation, and in particular, to systems and methods for automatically providing recommendations of optimized placement of objects with preset constraints in a home appliance (e.g., a dishwasher).
Conventional home appliances, such as dishwashers, rely on user manually loading the dishes. However, the user rarely pays attention to the instructions associated with the products to be washed or the instructions listed on the manual of the dishwasher to load the dishes. In fact, the way of loading dishes into a dishwasher can significantly impact the performance of the washing process in terms of washing quality, resource consumption (water, energy, soap, etc.), and time consumption. Sometimes, improper loading, such as loading dishes onto wrong racks (e.g., top or bottom), at wrong location on a rack, oriented in wrong directions, and/or loading dishwasher-unsafe objects into dishwasher may cause damages to the dishes and even cause malfunction to the dishwasher. For example, improper loading may clog water draining system, or cause parts to melt during drying process, thus affect the lifespan of the dishwasher. As such, these conventional home appliances require the user to read, understand, and practice the knowledge and or gain significant experience regarding how to properly load dishes into the dishwasher. Even if the user is willing to check the dishwasher manual or other online resources (e.g., images or videos that visually demonstrate how to load the dishwasher) to learn how to properly load the dishwasher, it is inconvenient for the user to hold and load the dishes while checking the manual or the online resources. Such process may be unintuitive, cumbersome, and interruptive, leaving the user frustrated and struggled to translate and follow the loading plan demonstrated from another media to loading the dishwasher in the real kitchen space.
For these reasons, improved methods and systems for assisting the user to load dishes on a rack of the dishwasher, and providing recommendations and guidance with such loading process are desirable.
Accordingly, there is a need for a method and system to assist the user with loading dishes properly and efficiently into the dishwasher, e.g., via a visual-aided recommendation system that provides intuitive and convenient visual or audio guidance and/or recommendation related to a proper dishwasher loading.
The embodiments described below provide systems and methods for providing visual-aided assistance to the user for properly load dishes in a dishwasher. The visual-aided assistance uses one or more built-in cameras installed on the dishwasher and/or one or more cameras on the user's mobile device to capture videos and/or images of the dishwasher chamber. Without requiring any user input to initiate the image capturing process or to request for recommendation or guidance for loading dishes into the dishwasher, the cameras can be initiated in response to various triggering events, such as when the system detects that the user opens the dishwasher door (e.g., indicating the user is going to load dishes on a rack in the dishwasher), when the system detects that the user pushes the rack back in (e.g., indicating that the user has completed loading the dishes and is about to start the washing process), and/or the system (e.g., the user's mobile device) detects that the user has launched the application on the user's mobile device.
After capturing the images, the placement recommendation system automatically starts to analyze the images to recognize one or more characteristics (e.g., size, shape, orientation, and/or materials) of the dishes, and generate recommendation and/or guidance regarding how to properly load dishes and/or how to correct the improper loading of one or more existing dishes on the rack. The recommendation and/or guidance can be provided via visual output (e.g., display on the dishwasher and/or on the user's mobile device display screen), audio output (e.g., speaker on the dishwasher and/or speaker of the user's mobile device), and/or other more intuitive and interactive visual cues in the dishwasher (e.g., shining a laser light towards the optimized location(s) on the rack to guide the user to place the dish on such location(s)). Thus, no additional user input is required to instruct the placement recommendation system to start performing the image analysis and running various algorithms as discussed herein to start performing image analysis and running various algorithms as discussed herein to generate recommendations. Such visual-aided placement recommendations can effectively and efficiently guide the user to properly load dishes in the dishwasher, thus avoiding damaging dishes and/or the dishwasher due to improper dish placement.
Further, the recommendation processes as discussed herein may not need any direct interaction between the user's hand and the dishwasher or the mobile phone (e.g., the camera is triggered by certain triggering events, and the recommendation analysis is automatically started following capturing the one or more images). This is convenient when the user's hands are occupied by the dishes, greasy from the dirty dishes, or inconvenient to interact with the dishwasher or the mobile phone. For example, the user may need loading suggestions while holding a dish in his or her hand, or when the user has greasy hands from the dirty dishes, or the user may be multi-tasking, checking his or her phone or watching TV and paying minimum attention to loading dishes in the dishwasher. The recommendation system discussed herein can automatically start the analysis and recommendation processes following the camera capturing the images without requiring any additional user input. In some embodiments, the camera capturing functions and/or the initiation of the recommendation processes may also controlled by user's voice input, thus completely freeing the user's hands from such tasks.
As disclosed herein, in some embodiments, a method of providing visual-aided placement recommendation is performed at a device (such as, a dishwasher or a mobile phone) having a camera and one or more output devices, one or more processors, and memory. The method includes obtaining one or more images of a rack configured to hold a plurality of objects in position while preset operations are performed on the plurality of objects inside a chamber, wherein placement of the plurality of objects on the rack are subject to one or more preset constraints corresponding to one or more characteristics of respective objects of the plurality of objects relative to one or more physical parameters of respective locations on the rack when the rack is placed within the chamber during the preset operations; analyzing the one or more images to determine whether the one or more preset constraints have been violated by placement of one or more objects on the rack; and in accordance with a determination that at least one of the one or more preset constraints has been violated by respective placement of at least a first object on the rack, generating a first output providing a guidance on proper placement of the first object on the rack that complies with the one or more preset constraints, wherein the first output is generated by the device in accordance with the one or more physical characteristics of the first object relative to the one or more physical parameters of the respective locations on the rack, taking into account of one or more other objects already placed on the rack.
In some embodiments, a method is performed at a device (such as, a dishwasher or a mobile phone) having a camera, one or more output devices, one or more processors, and memory. The method comprises detecting a triggering event for initiating the camera to capture one or more images of a mounting rack inside a chamber of a machine. In some embodiments, the mounting rack is configured to hold one or more objects at one or more locations on the mounting rack in accordance with one or more preset constraints (e.g., certain types of dishware are to be placed on top or bottom rack of the dishwasher, or dishware made of certain materials are not dishwasher-safe thus are not supposed to be placed in the dishwasher, or certain shape and/or size of dishware are supposed to be placed at locations on the rack that are designed to fit corresponding size and/or shape of dishes in the dishwasher. In some embodiments, the machine further includes a front door that when closed isolates the mounting rack and the chamber from outside of the machine. In some embodiments, the triggering event includes one or more of opening the front door of the machine; pushing the mounting rack from an extended position back into a retracted position inside the chamber; or detecting a user selection of a machine model from a listing of a plurality of machine models shown on the display in a graphical user interface. In some embodiments, in response to the triggering event, the method further includes initiating the camera on the device and capturing one or more images of the mounting rack. In some embodiments, the camera is initiated when the mounting rack is extended out and/or when the mounting rack is pushed back into the chamber. In some embodiments, the method further includes detecting one or more characteristics of an object based on the one or more images captured by the camera, wherein the one or more characteristics are used for determining a location and an orientation in accordance with the preset constraints to place the object on the mounting rack. In some embodiments, the characteristics of the object include shape, size, materials, orientation, etc. In some embodiments, the object is held by a user prior to placing the object on the mounting rack; or wherein the object is placed at a first location on the mounting rack. In accordance with the detecting one or more characteristics of the object, the method further includes identifying a first location and a first orientation for placing the object on the mounting rack according to the preset constraints; providing, on the display of the device, a notification associated with placing the object on the mounting rack in accordance with the identified first location and the first orientation. In some embodiments, the notification is provided in audio format, on the display, via a visual cue within the chamber of the device that points to the first location for placing the object. In some embodiments, the notification is a recommendation for placing the object at the first location in the first orientation; wherein the notification is an alert associated with a discrepancy between a current location and orientation of the object on the mounting rack and the first location and first orientation according to the preset constraints.
In some embodiments, a method is performed at a device (such as, a dishwasher) having (1) a mounting rack (e.g., extendable—can be extended for loading objects on the rack, and retracted after finishing loading) for holding one or more objects inside a chamber of the device at one or more locations respectively on the mounting rack according to one or more preset constraints and (2) a front door that when closed isolates the mounting rack and the chamber from an outside of the device. The device further includes one or more processors and memory. The method comprises detecting a first action of the front door (e.g., opening the front door); in response to detecting the first action of the front door, initiating a camera installed within the chamber of the device to capture one or more images to monitor/detect a movement of a user's hand within the chamber of the device; after initiating the camera, detecting a movement of a user's hand within the chamber and above the mounting rack of the device based on the captured one or more images; determining, in accordance with the one or more images of the movement of the user's hand captured by the camera, whether the movement of the user's hand corresponds to a moving-in action toward an inside of the chamber of the device (e.g., using a hand-tracking algorithm to analyze the images captured by the camera to determine whether a hand moving-in action by, for example, comparing a sequence of image frames to determine a moving direction of the user's hand); in accordance with a determination that the movement of the user's hand corresponds to a moving-in action toward the inside of the chamber, determining whether the user's hand is holding an object to place the object on the mounting rack within the chamber of the device (e.g., using a hand gesture analysis algorithm to analyze a hand gesture, such as holding an object in the user's hand). The method further includes: in accordance with determining that the user's hand is holding an object, detecting one or more characteristics (e.g., shape, size, material, etc. of the object) of the object based on the one or more images captured by the camera (e.g., using an object detection algorithm); prior to placing the object on the mounting rack, comparing the detected one or more characteristics of the object against the preset constraints for loading one or more objects on the mounting rack; in accordance with a comparison result, determining a first location on the mounting rack and a first orientation associated with placing the object on the mounting rack within the chamber of the device in accordance with the preset constraints (e.g., either an optimized location/orientation or a prohibited location/orientation); and in accordance with a determination result, providing a recommendation of the first location and the first orientation associated with placing the object on the mounting rack within the chamber of the device. In some embodiments, various recommendation formats can used, such as voice output, built-in display (e.g., a screen on the outside surface of the front door), sent to a user's mobile device, displaying a visual cue within the chamber of the device (e.g., shining a laser light towards the first location on the mounting rack).
In some embodiments, a method is performed at a device (such as, a dishwasher or a mobile phone) having (1) a mounting rack for holding one or more objects inside a chamber of the device at one or more locations respectively on the mounting rack according to preset constraints, (2) a front door that when closed isolates the mounting rack and the chamber from an outside of the device, (3) one or more processors, and (4) memory. The method comprises detecting a triggering event associated with an action of a part of the device (e.g., opening of the front door, or pushing the mounting rack from an extended position back in); in response to detecting the triggering event, initiating a camera installed within the device to capture one or more images of a top surface of the mounting rack within the chamber of the device; after initiating the camera, detecting one or more objects placed on the top surface of the mounting rack within the chamber and one or more characteristics of each detected object based on the one or more images captured by the camera (e.g., using an object detection algorithm); comparing the detected one or more characteristics of a respective object of the one or more objects against the preset constraints for loading objects on the mounting rack; in accordance with a comparison result, identifying a discrepancy in loading a first object on the mounting rack within the chamber of the device from the preset constraints; and providing an alert of the discrepancy and a recommendation of an optimized manner associated with loading the first object on the mounting rack within the chamber of the device in accordance with the preset constraints. In some embodiments, the recommendations can be provided in various formats, such as voice output, built-in display (e.g., a screen on the outside surface of the front door), display on a user's mobile device, and/or displaying a visual cue within the chamber of the device (e.g., shining a laser light towards the one or more optimized locations on the mounting rack).
In some embodiments, a method is performed at a mobile device having a camera, one or more output devices (e.g., a display), one or more processors, and memory. The method comprises: displaying, one the display, a listing of a plurality of device models in a graphical user interface, each device model corresponding to a respective device including (1) a mounting rack for holding one or more objects inside a chamber of the respective device at one or more locations respectively on the mounting rack according to a preset constraints and (2) a front door that when closed isolates the mounting rack and the chamber from an outside of the device; receiving a user selection of a first device model from the listing of the plurality of device models displayed in the graphical user interface; capturing, using the camera of the mobile device, an image of one or more objects placed on a top surface of the mounting rack within the chamber of a first device of the selected first device model; determining one or more characteristics of each of the one or more objects included in the image captured by the camera; comparing the detected one or more characteristics of a respective object of the one or more objects against the preset constraints for loading objects on the mounting rack; in accordance with a comparison result, identifying a discrepancy in loading a first object on the mounting rack within the chamber of the device from the preset constraints; providing an alert of the discrepancy and a recommendation of an optimized manner associated with loading the first object on the mounting rack within the chamber of the device in accordance with the preset constraints (e.g., in various recommendation formats, such as audio output on the mobile device and/or the dishwasher, visual output highlighting the optimized location(s) on the mobile device and/or a display of the dishwasher, haptic output such as vibration on mobile phone indicating there is a recommendation for dish placement to be checked on the phone and/or on the display of the dishwasher).
In accordance with some embodiments, a device (e.g., a dishwasher or a mobile phone) includes a camera, one or more output devices, one or more processors, and memory storing instruction, the instructions, when executed by the one or more processors, cause the processors to perform operations of any of the methods described herein. In accordance with some embodiments, a computer-readable storage medium (e.g., a non-transitory computer readable storage medium) is provided, the computer-readable storage medium storing one or more programs for execution by one or more processors of a voice control apparatus, the one or more programs including instructions for performing any of the methods described herein.
Various advantages of the present application are apparent in light of the descriptions below.
The aforementioned features and advantages of the disclosed technology, as well as additional features and advantages thereof, will be more clearly understood hereinafter as a result of a detailed description of preferred embodiments when taken in conjunction with the drawings.
To describe the technical solutions in the embodiments of the presently disclosed technology or in the prior art more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments or the prior art. Apparently, the accompanying drawings in the following description show merely some embodiments of the presently disclosed technology, and persons of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.
Like reference numerals refer to corresponding parts throughout the several views of the drawings.
Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the subject matter presented herein. But it will be apparent to one skilled in the art that the subject matter may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
The following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. The described embodiments are merely a part rather than all of the embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.
In some embodiments the one or more home appliances (e.g., smart dishwashers, smart ovens, smart microwave ovens, etc.) are configured to collect raw sensor data (e.g., image, weight, temperature, thermal map data, etc.) and send the raw sensor data to corresponding user devices (e.g., smart phones, tablet devices, etc.), and/or server system 120 (e.g., server provided by the manufacturer of the home appliances or third-party service providers for the manufacturer). In some embodiments, a home appliance is configured to receive control instructions from a control panel of the home appliance, server 120, and/or a corresponding user device. For example, the dishwasher 110 may receive control instructions from user interaction with a control panel and/or one or more buttons installed on the dishwasher for operating the dishwasher. The dishwasher 110 may also receive instructions from server system 120 with regard to optimized placement of dishes within the dishwasher based on images of the rack of the dishwasher. The dishwasher may further receive instructions from user device A 111 to capture one or more images of the dish mounting rack within the dishwasher chamber using the user device A 111. Additional details regarding the one or more home appliances (e.g., appliance A 110, appliance B 112, and appliance C 114) are described in detail with reference to other parts of the present disclosure.
In some embodiments, a respective appliance (e.g., the dishwasher 110) of the one or more home appliances includes an input/output user interface. The input/output user interface optionally includes one or more output devices that enable presentation of media content, including one or more speakers and/or one or more visual displays. The input/output user interface also optionally includes one or more input devices, including user interface components that facilitate user input, such as a keypad, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls.
In some embodiments, a respective appliance (e.g., the dishwasher 110) of the one or more home appliances further includes sensors, which senses environment information of the respective appliance. Sensors include but are not limited to one or more light sensors, cameras (also referred to as image sensors), humidity sensors, temperature sensors, motion sensors, weight sensors, spectrometers, and other sensors. In some embodiments, one or more devices and/or appliances in the operating environment 100 includes a respective camera and/or a respective motion sensor to detect presence of a user's hand and/or an appearance of an object (e.g., a dishware). In some embodiments, a camera install on the dishwasher 110 is used to capture one or more images associated with the dishwasher 110. For example, the camera is angled to detect motions of a user's hand inside the dishwasher chamber, and/or to monitor or capture images of dishes mounted on a rack inside the dishwasher chamber. In some embodiments, the sensors also provide information on the indoor environment, such as temperature, time of day, lighting, noise level, activity level of the room.
In some embodiments, the one or more user devices are configured to capture images related to dish mounting within the dishwasher chamber, receive raw sensor data from a respective appliance (e.g., user device A 111, which corresponds to appliance A 110, is configured to receive raw sensor data from appliance A 110), perform image analysis to evaluate dish placement within the dishwasher, and/or provide recommendations for optimized dish placement to improve the appliance performance and efficiency. In some embodiments, the one or more user devices are configured to generate and send control instructions to the respective appliance (e.g., user device A 111 may send instructions to appliance A 110 to turn appliance A 110 on/off or to capture one or more images of items placed on a rack of the appliance A 110).
In some embodiments, the one or more user devices include, but is not limited to, a mobile phone, a tablet, or a computer device. In some embodiments, more than one user device may correspond to one appliance (e.g., a computer and a mobile phone may both correspond to appliance A 110 (e.g., both are registered to be a control device for appliance A in an appliance setup process) such that appliance A 110 may send raw sensor data to either or both the computer and the mobile phone). In some embodiments, a user device corresponds to (e.g., shares data with and/or is in communication with) an appliance (e.g., user device A 111 corresponds to appliance A 110). For example, appliance A 110 may collect data (e.g., raw sensor data, such as images or temperature data) and send the collected data to user device A 111 so that the collected data may be annotated by a user on user device A 111.
In some embodiments, system server 120 is configured to receive raw sensor data from the one or more home appliances (e.g. appliances 110, 112, and 114), and/or receive annotated data from the one or more user devices (e.g., user devices 111, 113, and 115). In some embodiments, system server 120 is configured to received one or more images including dish placement on a mounting rack of the dishwasher 110, and/or hand motion associated with dish placement in the dishwasher 110. In some embodiments, system server 120 is configured to receive the captured images associated with the dishwasher 110 from camera(s) mounted on the dishwasher 110 and/or the user device 111. In some embodiments, system server 120 is configured to process the image data, recognize characteristics of the objects (e.g., dishware, cookware, etc.) placed on the mounting rack or held in the user hand to be placed on the mounting rack, and providing placement recommendation based on the processed image information and preset placement constraints of the dishwasher 110.
In some embodiments, home appliances (e.g. appliances 110, 112, and 114), user devices (e.g., user devices 111, 113, and 115), and server system 120 are connected (e.g., sharing data with and/or in communication with) through one or more networks 190. Examples of the communication network(s) 190 include local area networks (LAN) and wide area networks (WAN), e.g., the Internet. The communication network(s) 110 may be implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.
Examples of the user device 111 include, but are not limited to, a cellular telephone, a smart phone, a handheld computer, a wearable computing device (e.g., a HMD), a personal digital assistant (PDA), a tablet computer, a laptop computer, a desktop computer, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, a game console, a television, a remote control, a point of sale (POS) terminal, an e-book reader, a humanoid robot, or a combination of any two or more of these data processing devices or other data processing devices.
In some embodiments, user device 111 includes one or more of imaging processing module 155, network communication unit 136, and one or more databases 138. In some embodiments, user device 111 further includes user-side placement recommendation module 179 and user-side appliance-function control module 177 to facilitate the visual-aided placement recommendation and appliance control aspects of the system 101 as described herein.
In some embodiments, image processing module 155 obtains images captured by one or more cameras of the user device 111 and/or images captured by imaging system (e.g., image sensors 141,
In some embodiments, network communication unit 136 allows user device 111 to communicate with appliance 110 and/or system server 120 over one or more networks 190.
In some embodiments, databases 138 include a database of one or more preset constraints for placing objects in the home appliance 110 according to manufacturer settings or user customized settings. For example, the user device 111 may download the preset constraints for placing dishware in a dishwasher of a certain model. In some embodiments, the user may further edit, revise, or add additional constraints according to the user's need (e.g., as discussed with reference to
In some embodiments, an application running on user device 111 provides user-side functions, such as user-side placement recommendation and appliance-function control, in conjunction with system server 120 and appliance 110. In some embodiments, the application also provides access to contact the manufacturer or service providers for information and services related to the appliance 110.
In some embodiments, user-side placement recommendation module 149 is configured to provide recommendation for placing objects in the appliance in accordance with characteristics of the objects and preset constraints for placing objects on a rack inside the appliance.
In some embodiments, user-side placement recommendation module 149 is configured to automatically generate placement recommendation locally. In some embodiments, the user-side placement recommendation module 179 sends a request to system server 120, and receive the placement recommendation in real-time from the system server 120. The request includes real-time image data captured by appliance 110 or the user device 111, and the results are determined using characteristics of the objects to be placed on a rack in the appliance and preset constraints for placing objects on the rack that have been determined by manufacturer and/or customized by user.
In some embodiments, user-side appliance-function control module 177 is configured to provide a user interface for the user to directly control the appliance functions (e.g., turning the appliance on/off or setting an appliance parameter, etc.), and/or generate notifications based on the placement recommendation instruction. In some embodiments, placement recommendation is provided to the user-side appliance-function control module 177 from the user-side placement recommendation module 179.
In some embodiments, appliance 110 includes one or more first sensors (e.g., image sensor(s) 141), one or more washing units 143, display 144, I/O module 145, user interface 146, network communication unit 147, mechanical unit 148, control module 155, and, optionally, appliance-side placement recommendation module 149, and appliance-side appliance-function control unit 153. In some embodiments, the one or more devices and/or modules discussed herein are built-in units of the appliance 110. In some embodiments, one or more modules may be implemented on a computing device that is communicatively coupled with the appliance 110 to perform the corresponding functions as discussed herein.
In some embodiments, the image sensors 141 are mounted on the dishwasher and configured to capture images of the space including the rack for mounting dishes inside the dishwasher (e.g.,
Network communication unit 147 is analogous in function to network communication unit 137. Network communication unit 147 allows appliance 110 to communicate with user device 111 and/or system server 120 over one or more networks 190.
Mechanical unit 148 described herein refers to hardware and corresponding software and firmware components of appliance 110 that are configured to physically change the internal sensing (e.g., imaging), heating and/or washing configuration of the dishwasher 110.
In some embodiments, appliance-side placement recommendation module 149 includes functions that are similar to the user-side placement recommendation module 179. For example, the appliance-side placement recommendation module 149 is configured to provide recommendation regarding placement of objects within the dishwasher 110. For example, appliance-side placement recommendation module 149 is configured to, based on image data captured by image sensors 141, determine whether placement of one or more objects on the rack of the dishwasher includes mistakes. In some embodiments, appliance-side placement recommendation module 149 is configured to provide placement recommendation locally. In some embodiments, the appliance-side placement recommendation module 149 sends a request to system server 120, and receive the recommendations in real-time from the system server 120.
In some embodiments, the image sensors 141 are configured to capture unstructured image data. Examples of unstructured data include RGB images and thermal or infrared images. For example, the image sensors 141 may be configured to capture or record still images or videos of object placement on a rack of appliance 110. In some embodiments, imaging system 142 associated with the image sensors 141 includes a data storage system that stores the characteristics of objects to be placed on the rack, dimensions of the grooves and patterns of the rack, the distances between the camera and the rack, such that images taken by the cameras can be used to accurately determine the size and shape of the objects within the images.
In some embodiments, the image capturing is triggered when the image sensors detects that a user's hand holding an object to be placed on the rack enters into the field of view of the camera. In some embodiments, the image capturing is triggered when the dishwasher door is opened, indicating the user may be going to place one or more dishes into the dishwasher. In some embodiments, the image capturing is triggered when the dishwasher (e.g., by a sensor installed on the rack or vis images capture by the image sensors 141) detects that the rack is pushed in, indicating that the user has finished loading the dishes on the rack and is going to start the dishwasher, such that the recommendation system can check whether there needs to perform any adjustment of placement of the dishes on the rack. In some embodiments, the image capturing is triggered manually in response to a user's input, for example, the image capturing is triggered when the appliance 110 receives a user input received on the touch display or a button on the dishwasher 110, or the appliance 110 receives an instruction from the user device 111 which is generated in response to a user's instruction received on the user device 111 (e.g., via the application running on the user device 111) to inspect dish placement on the rack. Manual trigger is easier and less complicated to implement, and allows the user to purposefully capture images according to user's need to receive system recommendation regarding dish placement on the rack. In some embodiments, image processing module 161 obtains the images captured by the one or more image sensors 141, and preprocesses the images to remove the background from the images based on baseline images, e.g., captured before the insertion of dishes on the rack.
In some embodiments, control module 154 includes sensor control 162 configured to control and adjust the image sensors 141. For example, sensor control 162 may send instructions for the image sensors 141 to record videos (e.g., when tracking hand motion) or still images (e.g., after loading dishes capture an image for placement inspection).
In some embodiments, appliance-side appliance-function control module 153 is configured to control and adjust the various functions of appliance 110. For example, appliance-side appliance-function control module 153 may send instructions to washing units 143 to activate a first washing unit of the one or more heating units, or may send instructions to mechanical unit 148. In some embodiments, appliance-side appliance-function control module 153 generates and send control instructions to various components of the appliance 110 based on preconfigured operation protocols (e.g., to implement the normal routine functions of the appliance 110). In some embodiments, appliance-side appliance-function control module 153 generates and send control instructions to various components of the appliance 110 based on real-time dish-loading progress monitored by image sensors 141 within the appliance (e.g., to automatically provide guidance regarding where to place an object or an alert regarding wrong placement of the objects on the rack). In some embodiments, appliance-side appliance-function control module 153 generates and send control instructions to various components of the appliance 110 based on real-time user instructions received from user devices or via user interface 146 of appliance 110.
In some embodiments, the system server 120 is hosted by a manufacturer of the appliance 110. In some embodiments, the system server 120 includes one or more processing modules, such as image analysis module 172, server-side placement recommendation module 174, server-side appliance function control module 176, I/O interface to user device 111, I/O interface to appliance 110, an I/O interface to external services, and data and models stored in databases 178.
In some embodiments, the databases 178 store placement constraints for placing objects with different characteristics on the rack of a dishwasher of a certain model. The placement constraints may be defined by the manufacturer of the dishwasher and/or customized by the user. In some embodiments, the databases 178 further store appliance model information (e.g., dimensions of the mounting rack, distance between camera and the rack, etc.). In some embodiments, the databases further store user data such as information (e.g., dimensions) of user's cookware and dishware, customization of placement constraints, user preference of loading and/or using the dishwasher.
In some embodiments, the system server 120 communicates with external services (e.g., appliance manufacturer service(s), home appliance control service(s), navigation service(s), messaging service(s), information service(s), calendar services, social networking service(s), etc.) through the network(s) 240 for task completion or information acquisition. The I/O interface to the external services facilitates such communications. In some embodiments, the operation information (e.g., operation parameters, preset loading constraints) of the dishwasher is periodically updated at the system server 120 and transmitted to the appliance 110 and/or the user device 111 to store the updated information on the appliance 110 and/or user device 111 respectively. For example, as new shape and/or size of dishes made of new materials are available on the market or become the new trend or become the new household choice of the user, the preset constraints for placing the new dishes on the rack of the dishwasher are revised or updated, and such updated constraints are timely or periodically transmitted to the dishwasher and/or the mobile phone to update the corresponding information. In some embodiments, the image analysis module uses one or more machine learning models that are trained at the system server for hand motion tracking, rack monitoring, and/or object recognition, and the machine learning models are periodically updated at the system server and transmitted to the dishwasher and/or the user's mobile phone to update the corresponding imaging analyzing modules on the dishwasher and/or the mobile device.
In some embodiments, the image analysis module 172 stores hand tracking algorithm to track user's hand motion during an object placement process. The image analysis module 172 may stores object recognition algorithm for recognizing object characteristics, such as shape and size of the dishes to be mounted or already placed on the rack of the dishwasher. The image analysis module 172 may performs object placement inspection against preset constraints for placing objects on the rack of the dishwasher. In some embodiments, the server-side placement recommendation module 174 performs one or more functions similar to those performed by user-side placement recommendation module 179 and/or appliance-side placement recommendation module 149 as discussed herein. In some embodiments, the server-side appliance function control module 176 performs one or more function similar to those performed by user-side appliance-function control module 177 and/or appliance-side appliance function control module 153.
The functions of various systems within placement recommendation system 101 in
In some embodiments, the built-in camera or the camera of the user's device is used to monitor a dish loading process and further verify the dish loading layout on the rack (e.g., either an optimized loading layout prior to loading a certain model of the dishwasher, or a live loading situation during or after the user has placed dishes on the rack). In some embodiments, the placement recommendation system uses hand motion tracking algorithm and/or rack monitoring or tracking algorithm to easily and precisely identify the locations of the dishes on the rack. In some embodiments, for the dishwasher without built-in cameras, the user is required to hold the user device within a certain distance range from the rack and in a certain orientation to take images of the rack, so that the locations of the dishes or the grooves on the rack can be accurately identified in the images captured by the user device. In some embodiments as shown in the front view of the dishwasher 200 in
In some embodiments, the placement recommendation system includes hardware such as the built-in camera 204 on the dishwasher 200 or the camera on the user cell phone 206 communicatively coupled to the dishwasher 200 via the cloud-based computing system. In some embodiments, the hardware of the placement recommendation system further includes the display screen as an interface for displaying recommendations regarding dish placement to the user.
In some embodiments, the placement recommendation system includes software including hand detection algorithm for monitoring user's hand motion, objection detection algorithms for detecting dish types, materials, sizes, shapes, and locations on the rack, orientation algorithms for detecting dish orientation related to dish placement on the rack, and recommendation algorithm for identifying optimized locations for placing the detecting objects on the rack according to preset constraints.
In some embodiments, when the image processing module or the placement recommendation module finds that the moving-in hand is holding a dish, the placement recommendation system detects (308) characteristics, such as the type, material, size, and shape, of the dish included in the captured images. Based on the detection results, the placement recommendation system determines (310) the optimal rack, optimal location, optimal orientation to load the dish on the rack. In some embodiments, the placement recommendation system further provides (312) guidance or the recommendation of the proper placement of dishes on the rack. In some embodiments, the placement recommendation system provides the guidance or the recommendation in various ways such as voice output (314) via a built-in speaker of the dishwasher, visual output (316) to show text or highlighted map on a built-in display screen, or notifications displayed on a cellphone's app (318).
The advantage of using the process 500 in
In some embodiments, the user opens (602) an application running on the user's mobile phone. In some embodiments, the application is related to managing or operating the dishwasher. In some embodiments, the user selects (604) his or her dishwasher model from a list of dishwashers provided on the user interface of the app as shown in
In some embodiments as shown in
In some embodiments, for hand detection algorithm, dish detection and recognition algorithm, rack detection algorithms, we can adopt many popular algorithms depending the computation resource. In some embodiments, for the cloud based distributed systems when dishwashers can remotely access clouds, some highly computational algorithms (such as SSD, RetinaNet, MaskRCNN, etc.) can be used. In some embodiments, for the edge-based distributed systems where all the computations happen in edge, some low-cost and light-weight object detection algorithms such as MobileNet, SuffleNet. can be adopted. In some embodiments, for hand tracking module, algorithms such as CAMSHIFT, GOTURN, etc. can be used.
In some embodiments, for dishwashers with built-in camera and have high computation configurations, the processes 300 or 400 can be used. They can provide instant recommendations to the users. In some embodiments, for dish washers with built-in camera but have lower computation configurations, the process 500 can be used. In some embodiments, recommendations are provided only at the last step when the user is done with dish loading and right at the moment he pushes the rack in. In some embodiments, for dishwashers that do not have built-in camera, the process 600 can be used. The process 600 required the user to take pictures of the dish he is about to load and/or the current dish layout in the rack to get recommendations.
The examples discussed in
In some embodiments, criteria for triggering the camera to start capturing images of the rack, performing image analysis to guide the dish placement or to inform the user of any improper dish placement, and generating the recommendation alerts may be different depending on the stage of the dish loading and the information available. For example, prior to placing any dishes on the rack, the image capturing by the camera may be triggered by detection of the opening of the dishwasher door (e.g., as discussed with reference to
Further, the embodiment discussed in
In some embodiments, the method 700 includes obtaining (704) one or more images of a rack (e.g., a dish rack, or dish drawer) configured to hold a plurality of objects in position while preset operations (e.g., washing, scrubbing, rinsing, drying, disinfecting) are performed on the plurality of objects inside a chamber (e.g., a dishwasher chamber). In some embodiments, the plurality of objects include dishes, bowls, pots, pans, different kinds of utensils, wooden spoons, cups, glasses, water bottles, silicone molds, plastic containers, bakeware, knives, glass lids, aluminum pans. Although dish or dishware is use throughout the disclosure, it is to be understood that the objects to be loaded or being loaded in the dishwasher can include any type of objects as listed above. In some embodiments, placement of the plurality of objects on the rack are subject to one or more preset constraints corresponding to one or more characteristics (e.g., shape, size, orientation, materials, etc.) of respective objects of the plurality of objects relative to one or more physical parameters of respective locations on the rack. In some embodiments, such placement takes into consideration spatial configurations, such as dimensions and/or shapes of the rows, layers, wires, tines, baskets, and/or clips on the rack, when the rack is placed within the chamber during the preset operations. In some embodiments, some locations that appear to hold the dishes fine when the rack is outside will not work because they may not be accessible by sprayed water, or may block a spray arm from swinging, may block other objects from being sprayed and cleaned, or may be blocked by other parts of the dishwasher chamber once the rack is inserted or in the process of bring inserted into position in the dishwasher chamber, or the height of the dishes may not fit once the rack is placed inside the dishwasher chamber.
In some embodiments, the method 700 includes analyzing (706) the one or more images to determine whether the one or more preset constraints have been violated by placement of one or more objects on the rack. For example, the one or more images may be analyzed using one or more algorithms, such as hand motion tracking algorithm, rack monitoring algorithm, object recognition algorithm, as discussed herein.
In some embodiments, the method 700 includes generating (708) a first output providing a guidance on proper placement of the first object on the rack that complies with the one or more preset constraints, in accordance with a determination that at least one of the one or more preset constraints has been violated by respective placement of at least a first object on the rack. In some embodiments, this violation can be detected during placement of the first object or after placement of all objects. In some embodiments, the first output is generated by the device in accordance with the one or more physical characteristics of the first object relative to the one or more physical parameters of the respective locations on the rack, taking into account of one or more other objects already placed on the rack. In some embodiments, the first output is generated depending on the quantity, locations, and physical characteristics of other objects already loaded onto the rack at the time that the first object is being loaded onto the rack. For example, certain locations on the rack may be blocked (e.g., blocked by other objects already loaded, objects that are later loaded, and/or internal parts of the chamber), unavailable (e.g., occupied by other objects that are already loaded, objects that are later loaded, or internal parts of the chamber (e.g., sprayer arm, soap dispenser, etc.)), unsuitable for the first object (e.g., unsuitable temperature at the bottom rack for the first object) once the rack is pushed into the dish washer chamber).
In some embodiments, the rack is part of a dishwasher that performs preset dish washing operations on the plurality of objects when the rack is inserted into the dishwasher. In some embodiments, the one or more preset constraints include a first constraint imposed by a position of a sprayer of the dishwasher, such as a fixed sprayer, or a moveable sprayer (e.g., with lateral movement or rotational movement during performance the preset operations) inside the chamber.
In some embodiments, the one or more preset constraints include a second constraint imposed by temperature distribution within the chamber during performance of the preset operations. For example, upper portion of the chamber (e.g., a top rack) has lower temperatures and are suitable for plastic objects, and lower portion of the chamber (e.g., a bottom rack) has higher temperatures and are suitable for metal, glass, and ceramic objects.
In some embodiments, the one or more preset constraints include a third constraint imposed by presence of an earlier loaded object that faces a first direction relative to a preset portion of the rack. For example, the earlier loaded object (e.g., a dish) faces toward the center of the rack, and the third constraint requires an adjacent object to also face the same direction as the earlier loaded object.
In some embodiments, the one or more preset constraints include a fourth constraint that prevents a concave surface of a respective object to face upward in the chamber. For example, a dish or bowl should not be faced upward during dish washing.
In some embodiments, the device is a dishwasher (e.g., the appliance 110 of
In some embodiments, the device is a mobile device (e.g., user device 111 of
In some embodiments, obtaining the one or more images of the rack configured to hold a plurality of objects in position while preset operations are performed on the plurality of objects inside a chamber is performed in response to detecting opening of the chamber. For example, the images are obtained in response to detecting opening of a front door of the dishwasher (e.g.,
In some embodiments, obtaining the one or more images of the rack configured to hold a plurality of objects in position while preset operations are performed on the plurality of objects inside a chamber is performed in response to detecting movement of an object toward the rack after the chamber is opened. In some embodiments, the images are obtained in response to detecting a hand moving toward the rack, and the hand motion is further tracked. Then, the method further includes identifying the object held by the hand as the hand is moving toward the rack. For example, the device (e.g., dishwasher) includes a actuation sensor attached to the door of the device or the rack of the device, and movement of the door or the rack generates a trigger event that triggers the camera attached to the front of the device (e.g., behind the door on the door frame of the dishwasher) to capture a video, and each time an new object is being brought toward the rack, the device starts a new analysis for the currently held object based on a segment of the video corresponding to the loading of the current object.
In some embodiments, obtaining the one or more images of the rack configured to hold a plurality of objects in position while preset operations are performed on the plurality of objects inside a chamber is performed in response to detecting movement of the rack into the chamber. For example, the device (e.g., dishwasher) includes a actuation sensor attached to the door of the device or the rack of the device, and movement of the rack into the chamber of the device generates a trigger event that triggers the camera attached to the front of the device (e.g., behind the door on the door frame of the dishwasher) to capture one or more images that show the rack after the loading of all objects.
In some embodiments, obtaining the one or more images of the rack is performed during loading of a respective object onto the rack and after completion of loading for a respective object load. For example, placement of each object is assessed during the loading of that object, and location/placement recommendation is provided for the object; and after placement of all objects for a load, alerts are provided if it is determined that a respective object violates the preset constraints in light of other objects placed in the rack at the completion of the loading.
In some embodiments, different sets of criteria are used for generating the first output depending on whether the violation is found during loading of the respective object or after the completion of the loading. For example, during loading of a respective object, more stringent criteria are used to determine whether the one or more constraints are violated (e.g., proper loading is defined by optimal or recommended loading practice guidelines), and recommendations are provided if the predicted loading location/orientation (e.g., based on the direction of movement of the hand, and how the object is being held) or initial location/orientation of the object are less than optimal based on the currently loaded objects and their distributions in the rack. After the completion of loading, all the objects are loaded into the rack, and a less stringent set of requirements are used to determine if the objects were loaded properly. For example, the loading is considered proper as long as the dishwasher will function properly, e.g., sprayer will reach all regions (e.g., maybe not with all equal efficiency), detergent dispenser door will open, and sprayer will not blocked during operation.
In some embodiments, the first output includes a first component that provides a recommendation or error correction instruction for the proper placement of the first object on the rack, and an explanation for the recommendation or error correction instruction. For example, the error or recommendation for a specific object is used as an example for a teaching moment to educate the user about how a dishwasher should be loaded, so that the user can do better next time when other similar objects are being loaded into the dish washer.
In some embodiments, analyzing the one or more images to determine whether the one or more preset constraints have been violated by placement of one or more objects on the rack include optimizing one or more performance parameters (e.g., optimal distribution of detergent, water, and temperature, cleansing efficiency, energy efficiency, water efficiency, etc.) associated with the preset operations by adjusting a location and orientation of a respective object on the rack and checking against one or more fixed rules regarding a location and orientation of the respective object on the rack. For example, the dishes can't block sprayer, while must be accessible by water and detergent for adequate cleaning and rinsing. In another example, the dishes must allow proper drainage of water, must allow opening of detergent dispenser, must allow proper closing of the door, and stability of the whole machine.
In some embodiments, the method 700 further includes obtaining an image of all objects that are to be loaded onto the rack from a mobile device prior to obtaining the one or more images. For example, the user takes a photo of all dishes that need to be loaded into the dishwasher using a mobile phone, and transmits the photo to the dishwasher before she starts loading the dishwasher. In some embodiments, the first output is generated by the device further in accordance analysis taking into account other objects among all the objects that are to be loaded onto the rack but that have not already placed on the rack.
In some embodiments, although the camera on the dishwasher and/or on the mobile phone is triggered to start capturing images in response to one or more triggering events (e.g., the user opens the dishwasher door, the user pushes the rack back into the dishwasher chamber, etc.), the placement recommendation system is automatically started to analyze the captured images, identify improper dish placement, identify optimized location(s) for placing the dish(es), and provide the guidance or recommendation to the user via one or more output formats as discussed herein. That is, the user may not need to use additional user input to instruct the placement recommendation system to analyze the
The recommendation processes as discussed herein may not need any direct interaction between the user's hand and the dishwasher or the mobile phone (e.g., the camera is triggered by certain triggering events, and the recommendation analysis is automatically started following capturing the one or more images). This is convenient when the user's hands are occupied or inconvenient to interact with the dishwasher or the mobile phone, for example, when the user is holding a dish in his or her hand, or when the user has greasy hands from cooking, the user may not want to touch the dishwasher or the mobile phone. The recommendation system can automatically start the analysis and recommendation processes following the camera capturing the images without requiring any additional user input. In some embodiments, the camera capturing functions and/or the initiation of the recommendation processes may also controlled by user's voice input, thus freeing the user's busy hands occupied from errands in the kitchen.
Memory 806 includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory 806, optionally, includes one or more storage devices remotely located from one or more processing units 802. Memory 806, or alternatively the non-volatile memory within memory 806, includes a non-transitory computer readable storage medium. In some implementations, memory 806, or the non-transitory computer readable storage medium of memory 806, stores the following programs, modules, and data structures, or a subset or superset thereof:
Memory 956 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 956, optionally, includes one or more storage devices remotely located from one or more processing units 952. Memory 956, or alternatively the non-volatile memory within memory 956, includes a non-transitory computer readable storage medium. In some implementations, memory 956, or the non-transitory computer readable storage medium of memory 956, stores the following programs, modules, and data structures, or a subset or superset thereof:
Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, modules or data structures, and thus various subsets of these modules may be combined or otherwise re-arranged in various implementations. In some implementations, memory 506, optionally, stores a subset of the modules and data structures identified above. Furthermore, memory 506, optionally, stores additional modules and data structures not described above.
While particular embodiments are described above, it will be understood it is not intended to limit the application to these particular embodiments. On the contrary, the application includes alternatives, modifications and equivalents that are within the spirit and scope of the appended claims. Numerous specific details are set forth in order to provide a thorough understanding of the subject matter presented herein. But it will be apparent to one of ordinary skill in the art that the subject matter may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
