Patent Application
20230374715
The present subject matter relates generally to washing machine appliances, and more particularly to methods for operating washing machine appliances to avoid color transfer within a load of clothes.
Washing machine appliances generally include a cabinet that receives a tub for containing wash and rinse water. A wash basket is rotatably mounted within the tub. A drive assembly is coupled to the tub and configured to rotate the wash basket within the tub in order to cleanse articles within the wash basket. Upon completion of a wash cycle, a pump assembly can be used to rinse and drain soiled water to a draining system. Some washing machine appliances may also rotate the wash basket at a relatively high speed for a spin cycle to further drain or shed water from articles within the wash basket.
Prior to an operating cycle, a user typically places a load of laundry in the wash chamber, selects cycle parameters, and initiates the wash cycle. However, if a user loads the wash chamber with clothes having different colors, it is possible that initiating the wash cycle may result in colors bleeding among the clothes. For example, if a user provides a load that is primarily bright whites but includes a dark item as well, e.g., such as jeans or a dark sweater, the load of bright whites may be contaminated with dye or color that bleeds from the dark item. This is particularly true for new items that have not been through many wash cycles. Notably, conventional washing machine appliances do not have methods for detecting load conditions that may result in color contamination, e.g., when a user inadvertently loads a light-colored garment in a dark load or a dark-colored garment in a light load.
Accordingly, a washing machine appliance with improved systems and methods for preventing color contamination within loads is desirable. More specifically, a method for automatically detecting situations where color bleed may occur and implementing correction action would be particularly beneficial.
Aspects and advantages of the invention will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the invention.
In one exemplary embodiment, a washing machine appliance is provided including a wash tub positioned within a cabinet, a wash basket rotatably mounted within the wash tub and defining a wash chamber configured for receiving a load of clothes, a camera assembly positioned in view of the load of clothes, and a controller operably coupled to the camera assembly. The controller is configured to obtain one or more images of the load of clothes to be washed during a wash cycle, analyze the one or more images using an image recognition process to identify an outlier garment in the load of clothes, determine that the outlier garment is not on a safelist, and implement a responsive action in response to identifying the outlier garment that is not on the safelist.
In another exemplary embodiment, a method of operating a washing machine appliance is provided. The washing machine appliance includes a wash basket rotatably mounted within a wash tub and defining a wash chamber configured for receiving a load of clothes, and a camera assembly positioned in view of the load of clothes. The method includes obtaining one or more images of the load of clothes to be washed during a wash cycle, analyzing the one or more images using an image recognition process to identify an outlier garment in the load of clothes, determining that the outlier garment is not on a safelist, and implementing a responsive action in response to identifying the outlier garment that is not on the safelist.
These and other features, aspects and advantages of the present invention will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
A full and enabling disclosure of the present invention, including the best mode thereof, directed to one of ordinary skill in the art, is set forth in the specification, which makes reference to the appended figures.
Repeat use of reference characters in the present specification and drawings is intended to represent the same or analogous features or elements of the present invention.
Reference now will be made in detail to embodiments of the invention, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the invention, not limitation of the invention. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the scope or spirit of the invention. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present invention covers such modifications and variations as come within the scope of the appended claims and their equivalents.
As used herein, the terms “first,” “second,” and “third” may be used interchangeably to distinguish one component from another and are not intended to signify location or importance of the individual components. The terms “includes” and “including” are intended to be inclusive in a manner similar to the term “comprising.” Similarly, the term “or” is generally intended to be inclusive (i.e., “A or B” is intended to mean “A or B or both”). In addition, here and throughout the specification and claims, range limitations may be combined and/or interchanged. Such ranges are identified and include all the sub-ranges contained therein unless context or language indicates otherwise. For example, all ranges disclosed herein are inclusive of the endpoints, and the endpoints are independently combinable with each other. The singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.
Approximating language, as used herein throughout the specification and claims, may be applied to modify any quantitative representation that could permissibly vary without resulting in a change in the basic function to which it is related. Accordingly, a value modified by a term or terms, such as “generally,” “about,” “approximately,” and “substantially,” are not to be limited to the precise value specified. In at least some instances, the approximating language may correspond to the precision of an instrument for measuring the value, or the precision of the methods or machines for constructing or manufacturing the components and/or systems. For example, the approximating language may refer to being within a 10 percent margin, i.e., including values within ten percent greater or less than the stated value. In this regard, for example, when used in the context of an angle or direction, such terms include within ten degrees greater or less than the stated angle or direction, e.g., “generally vertical” includes forming an angle of up to ten degrees in any direction, e.g., clockwise or counterclockwise, with the vertical direction V.
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” In addition, references to “an embodiment” or “one embodiment” does not necessarily refer to the same embodiment, although it may. Any implementation described herein as “exemplary” or “an embodiment” is not necessarily to be construed as preferred or advantageous over other implementations. Moreover, each example is provided by way of explanation of the invention, not limitation of the invention. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the scope of the invention. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present invention covers such modifications and variations as come within the scope of the appended claims and their equivalents.
Referring now to the figures, an exemplary laundry appliance that may be used to implement aspects of the present subject matter will be described. Specifically,
According to exemplary embodiments, washing machine appliance 100 includes a cabinet 102 that is generally configured for containing and/or supporting various components of washing machine appliance 100 and which may also define one or more internal chambers or compartments of washing machine appliance 100. In this regard, as used herein, the terms “cabinet,” “housing,” and the like are generally intended to refer to an outer frame or support structure for washing machine appliance 100, e.g., including any suitable number, type, and configuration of support structures formed from any suitable materials, such as a system of elongated support members, a plurality of interconnected panels, or some combination thereof. It should be appreciated that cabinet 102 does not necessarily require an enclosure and may simply include open structure supporting various elements of washing machine appliance 100. By contrast, cabinet 102 may enclose some or all portions of an interior of cabinet 102. It should be appreciated that cabinet 102 may have any suitable size, shape, and configuration while remaining within the scope of the present subject matter.
As illustrated, cabinet 102 generally extends between a top 104 and a bottom 106 along the vertical direction V, between a first side 108 (e.g., the left side when viewed from the front as in
Referring to
Wash basket 120 may define one or more agitator features that extend into wash chamber 126 to assist in agitation and cleaning articles disposed within wash chamber 126 during operation of washing machine appliance 100. For example, as illustrated in
According to exemplary embodiments, wash tub 124 may be generally suspended within cabinet 102 by one or more suspension assemblies 129, e.g., as shown for example in
Referring generally to
A window 136 in door 134 permits viewing of wash basket 120 when door 134 is in the closed position, e.g., during operation of washing machine appliance 100. Door 134 also includes a handle (not shown) that, e.g., a user may pull when opening and closing door 134. Further, although door 134 is illustrated as mounted to front panel 130, it should be appreciated that door 134 may be mounted to another side of cabinet 102 or any other suitable support according to alternative embodiments. Washing machine appliance 100 may further include a latch assembly 138 (see
Referring again to
A drain pump assembly 144 is located beneath wash tub 124 and is in fluid communication with sump 142 for periodically discharging soiled wash fluid from washing machine appliance 100. Drain pump assembly 144 may generally include a drain pump 146 which is in fluid communication with sump 142 and with an external drain 148 through a drain hose 150. During a drain cycle, drain pump 146 urges a flow of wash fluid from sump 142, through drain hose 150, and to external drain 148. More specifically, drain pump 146 includes a motor (not shown) which is energized during a drain cycle such that drain pump 146 draws wash fluid from sump 142 and urges it through drain hose 150 to external drain 148.
Washing machine appliance 100 may further include a wash fluid dispenser that is generally configured for dispensing a flow of water, wash fluid, etc. into wash tub 124. For example, a spout 152 is configured for directing a flow of fluid into wash tub 124. For example, spout 152 may be in fluid communication with a water supply 155 (
As illustrated in
In addition, a water supply valve 158 may provide a flow of water from a water supply source (such as a municipal water supply 155) into detergent dispenser 156 and into wash tub 124. In this manner, water supply valve 158 may generally be operable to supply water into detergent dispenser 156 to generate a wash fluid, e.g., for use in a wash cycle, or a flow of fresh water, e.g., for a rinse cycle. It should be appreciated that water supply valve 158 may be positioned at any other suitable location within cabinet 102. In addition, although water supply valve 158 is described herein as regulating the flow of “wash fluid,” it should be appreciated that this term includes, water, detergent, other additives, or some mixture thereof.
During operation of washing machine appliance 100, laundry items are loaded into wash basket 120 through opening 132, and washing operation is initiated through operator manipulation of one or more input selectors or using a remote device (see below). Wash tub 124 is filled with water, detergent, and/or other fluid additives, e.g., via spout 152 and/or detergent drawer 156. One or more valves (e.g., water supply valve 158) can be controlled by washing machine appliance 100 to provide for filling wash basket 120 to the appropriate level for the amount of articles being washed and/or rinsed. By way of example for a wash mode, once wash basket 120 is properly filled with fluid, the contents of wash basket 120 can be agitated (e.g., with ribs 128) for washing of laundry items in wash basket 120.
After the agitation phase of the wash cycle is completed, wash tub 124 can be drained. Laundry articles can then be rinsed by again adding fluid to wash tub 124, depending on the particulars of the cleaning cycle selected by a user. Ribs 128 may again provide agitation within wash basket 120. One or more spin cycles may also be used. In particular, a spin cycle may be applied after the wash cycle and/or after the rinse cycle in order to wring wash fluid from the articles being washed. During a final spin cycle, basket 120 is rotated at relatively high speeds and drain assembly 144 may discharge wash fluid from sump 142. After articles disposed in wash basket 120 are cleaned, washed, and/or rinsed, the user can remove the articles from wash basket 120, e.g., by opening door 134 and reaching into wash basket 120 through opening 132.
Referring again to
Washing machine appliance 100 may further include or be in operative communication with a processing device or a controller 166 that may be generally configured to facilitate appliance operation. In this regard, control panel 160, user input devices 162, and display 164 may be in communication with controller 166 such that controller 166 may receive control inputs from user input devices 162, may display information using display 164, and may otherwise regulate operation of washing machine appliance 100. For example, signals generated by controller 166 may operate washing machine appliance 100, including any or all system components, subsystems, or interconnected devices, in response to the position of user input devices 162 and other control commands. Control panel 160 and other components of washing machine appliance 100 may be in communication with controller 166 via, for example, one or more signal lines or shared communication busses. In this manner, Input/Output (“I/O”) signals may be routed between controller 166 and various operational components of washing machine appliance 100.
As used herein, the terms “processing device,” “computing device,” “controller,” or the like may generally refer to any suitable processing device, such as a general or special purpose microprocessor, a microcontroller, an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a logic device, one or more central processing units (CPUs), a graphics processing units (GPUs), processing units performing other specialized calculations, semiconductor devices, etc. In addition, these “controllers” are not necessarily restricted to a single element but may include any suitable number, type, and configuration of processing devices integrated in any suitable manner to facilitate appliance operation. Alternatively, controller 166 may be constructed without using a microprocessor, e.g., using a combination of discrete analog and/or digital logic circuitry (such as switches, amplifiers, integrators, comparators, flip-flops, AND/OR gates, and the like) to perform control functionality instead of relying upon software.
Controller 166 may include, or be associated with, one or more memory elements or non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, or other suitable memory devices (including combinations thereof). These memory devices may be a separate component from the processor or may be included onboard within the processor. In addition, these memory devices can store information and/or data accessible by the one or more processors, including instructions that can be executed by the one or more processors. It should be appreciated that the instructions can be software written in any suitable programming language or can be implemented in hardware. Additionally, or alternatively, the instructions can be executed logically and/or virtually using separate threads on one or more processors.
For example, controller 166 may be operable to execute programming instructions or micro-control code associated with an operating cycle of washing machine appliance 100. In this regard, the instructions may be software or any set of instructions that when executed by the processing device, cause the processing device to perform operations, such as running one or more software applications, displaying a user interface, receiving user input, processing user input, etc. Moreover, it should be noted that controller 166 as disclosed herein is capable of and may be operable to perform any methods, method steps, or portions of methods of appliance operation. For example, in some embodiments, these methods may be embodied in programming instructions stored in the memory and executed by controller 166.
The memory devices may also store data that can be retrieved, manipulated, created, or stored by the one or more processors or portions of controller 166. The data can include, for instance, data to facilitate performance of methods described herein. The data can be stored locally (e.g., on controller 166) in one or more databases and/or may be split up so that the data is stored in multiple locations. In addition, or alternatively, the one or more database(s) can be connected to controller 166 through any suitable network(s), such as through a high bandwidth local area network (LAN) or wide area network (WAN). In this regard, for example, controller 166 may further include a communication module or interface that may be used to communicate with one or more other component(s) of washing machine appliance 100, controller 166, an external appliance controller, or any other suitable device, e.g., via any suitable communication lines or network(s) and using any suitable communication protocol. The communication interface can include any suitable components for interfacing with one or more network(s), including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.
Referring now specifically to
Although an exemplary camera assembly 170 is illustrated and described herein, it should be appreciated that according to alternative embodiments, washing machine appliance 100 may include any other camera or system of imaging devices for obtaining images of the load of clothes 172, wash chamber 126, etc. It should be appreciated that camera assembly 170 may include any suitable number, type, size, and configuration of camera(s) 178 for obtaining images of wash chamber 126. In general, cameras 178 may include a lens that is constructed from a clear hydrophobic material or which may otherwise be positioned behind a hydrophobic clear lens. So positioned, camera assembly 170 may obtain one or more images or videos of clothes 172 within wash chamber 126, as described in more detail below. Washing machine appliance 100 may further include a tub light that is positioned within cabinet 102 or wash chamber 126 for selectively illuminating wash chamber 126 and/or the load of clothes 172 positioned therein.
Referring again to
For example, external communication system 180 permits controller 166 of washing machine appliance 100 to communicate with a separate device external to washing machine appliance 100, referred to generally herein as an external device 182. As described in more detail below, these communications may be facilitated using a wired or wireless connection, such as via a network 184. In general, external device 182 may be any suitable device separate from washing machine appliance 100 that is configured to provide and/or receive communications, information, data, or commands from a user. In this regard, external device 182 may be, for example, a personal phone, a smartphone, a tablet, a laptop or personal computer, a wearable device, a smart home system, or another mobile or remote device.
In addition, a remote server 186 may be in communication with washing machine appliance 100 and/or external device 182 through network 184. In this regard, for example, remote server 186 may be a cloud-based server 186, and is thus located at a distant location, such as in a separate state, country, etc. According to an exemplary embodiment, external device 182 may communicate with a remote server 186 over network 184, such as the Internet, to transmit/receive data or information, provide user inputs, receive user notifications or instructions, interact with or control washing machine appliance 100, etc. In addition, external device 182 and remote server 186 may communicate with washing machine appliance 100 to communicate similar information.
In general, communication between washing machine appliance 100, external device 182, remote server 186, and/or other user devices or appliances may be carried using any type of wired or wireless connection and using any suitable type of communication network, non-limiting examples of which are provided below. For example, external device 182 may be in direct or indirect communication with washing machine appliance 100 through any suitable wired or wireless communication connections or interfaces, such as network 184. For example, network 184 may include one or more of a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, a cellular network, any other suitable short- or long-range wireless networks, etc. In addition, communications may be transmitted using any suitable communications devices or protocols, such as via Wi-Fi®, Bluetooth®, Zigbee®, wireless radio, laser, infrared, Ethernet type devices and interfaces, etc. In addition, such communication may use a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).
External communication system 180 is described herein according to an exemplary embodiment of the present subject matter. However, it should be appreciated that the exemplary functions and configurations of external communication system 180 provided herein are used only as examples to facilitate description of aspects of the present subject matter. System configurations may vary, other communication devices may be used to communicate directly or indirectly with one or more associated appliances, other communication protocols and steps may be implemented, etc. These variations and modifications are contemplated as within the scope of the present subject matter.
While described in the context of a specific embodiment of horizontal axis washing machine appliance 100, using the teachings disclosed herein it will be understood that horizontal axis washing machine appliance 100 is provided by way of example only. Other washing machine appliances having different configurations, different appearances, and/or different features may also be utilized with the present subject matter as well, e.g., vertical axis washing machine appliances.
Now that the construction of washing machine appliance 100 and the configuration of controller 166 according to exemplary embodiments have been presented, an exemplary method 200 of operating a washing machine appliance will be described. Although the discussion below refers to the exemplary method 200 of operating washing machine appliance 100, one skilled in the art will appreciate that the exemplary method 200 is applicable to the operation of a variety of other washing machine appliances, such as vertical axis washing machine appliances. In exemplary embodiments, the various method steps as disclosed herein may be performed by controller 166 or a separate, dedicated controller.
Referring now to
Although step 210 is described herein as obtaining images of the load of clothes, it should be appreciated that obtaining these images may include obtaining more than one image, a series of frames, a video, or any other suitable visual representation of the load of clothes 172 using camera assembly 170. In addition, it should be appreciated that the images obtained by camera assembly 170 may vary in number, frequency, angle, resolution, detail, etc. in order to improve the clarity of the load of clothes 172. In addition, according to exemplary embodiments, controller 166 may be configured for illuminating the tub or the load of clothes 172 using a tub light just prior to obtaining images. The obtaining images may also be cropped in any suitable manner for improved focus on desired portions of the load of clothes 172.
The one or more images may be obtained using camera assembly 170 at any suitable time prior to initiating the wash cycle. For example, as best illustrated in
According to still other embodiments, as shown for example in
Accordingly, method 200 may include tumbling the load of clothes 172 between images to obtain a better visual representation of the entire load of clothes, e.g., to better identify color contamination issues and outlier garments. In this regard, method 200 may include operating motor assembly 122 to rotate wash basket 120 to tumble the load of clothes 172 within wash chamber 126. It should be appreciated that any suitable agitation profile, intensity, and duration may be used to tumble the load of clothes. For example, motor assembly 122 may be operated until a turnover condition of the load of clothes has been satisfied (e.g., rotation has been sufficient to reposition or shuffle the load of clothes within wash basket 120), after which the motor assembly may be stopped and additional images may be taken.
As explained in more detail below, the one or more images may be used to monitor the load of clothes and identify one or more outlier garments. As used herein, the terms “outlier garment” and the like may be used generally refer to any items of clothing or other objects that are present within a load of clothes and present a risk of color contamination of the load of clothes. For example, if the load of clothes is primarily white shirts or bright whites and blue jeans are present within the load of clothes, color bleed from the blue jeans may permanently stain or discolor the lighter colored garments within the load. Accordingly, the blue jeans may be referred to as an outlier garment within the load.
Referring still to
According to exemplary embodiments, the image analysis may use any suitable image processing technique, image recognition process, etc. As used herein, the terms “image analysis” and the like may be used generally to refer to any suitable method of observation, analysis, image decomposition, feature extraction, image classification, etc. of one or more images, videos, or other visual representations of an object. As explained in more detail below, this image analysis may include the implementation of image processing techniques, image recognition techniques, or any suitable combination thereof. In this regard, the image analysis may use any suitable image analysis software or algorithm to constantly or periodically monitor the load of clothes 172 or the garments contained therein. It should be appreciated that this image analysis or processing may be performed locally (e.g., by controller 166) or remotely (e.g., by offloading image data to a remote server or network, e.g., remote server 186).
Specifically, the analysis of the one or more images may include implementation of an image processing algorithm. As used herein, the terms “image processing” and the like are generally intended to refer to any suitable methods or algorithms for analyzing images that do not rely on artificial intelligence or machine learning techniques (e.g., in contrast to the machine learning image recognition processes described below). For example, the image processing algorithm may rely on image differentiation, e.g., such as a pixel-by-pixel comparison of two sequential images. This comparison may help identify substantial differences between the sequentially obtained images, e.g., to identify the perimeter of garments, to identify colors or color differences within the load of clothes 172, etc. For example, one or more reference images may be obtained of a particular garment or garment color, and these references images may be stored for future comparison with images obtained during appliance operation. Similarities and/or differences between the reference image and the obtained image may be used to extract useful information for improving appliance performance.
According to exemplary embodiments, the image analysis performed at step 220 may generally include generating or preparing a color histogram of the images. In this regard, color histogram may generally include a representation of the distribution of colors with an image. For example, the color histogram may include a number of pixels within each image that have colors within a specific range. After preparing the color histogram for each image, the pixel color identification may be compared to a predetermined color ranges or thresholds, e.g., such as ranges associated with dark items, light items, white items, etc. By comparing the pixels from the color histogram with predetermined color values, outlier garments may be identified.
Notably, it should be appreciated that outlier garments may generally be defined relative to the remainder of the load. In this regard, a dark black sock may be an outlier garment when placed within a load of white shirts but may not be an outlier garment when placed within a load of dark gray pants. According to exemplary embodiments, a difference between the darkness level of the potential outlier garment and an average darkness level of the remainder of the load may be used to determine whether responsive action should be taken. In this regard, for example, analyzing the images may include identifying and outlier darkness level or outlier garment color of the one or more outlier garments and a load darkness level or primary load color of the remainder of the load of clothes. This analysis may further include determining that a difference between the outlier garment color and the primary load color exceeds a predetermined threshold.
According to exemplary embodiments, image processing may include blur detection algorithms that are generally intended to compute, measure, or otherwise determine the amount of blur in an image. For example, these blur detection algorithms may rely on focus measure operators, the Fast Fourier Transform along with examination of the frequency distributions, determining the variance of a Laplacian operator, or any other methods of blur detection known by those having ordinary skill in the art. In addition, or alternatively, the image processing algorithms may use other suitable techniques for recognizing or identifying items or objects, such as edge matching or detection, divide-and-conquer searching, greyscale matching, histograms of receptive field responses, or another suitable routine (e.g., executed at the controller 166 based on one or more captured images from one or more cameras). Other image processing techniques are possible and within the scope of the present subject matter. The processing algorithm may further include measures for isolating or eliminating noise in the image comparison, e.g., due to image resolution, data transmission errors, inconsistent lighting, or other imaging errors. By eliminating such noise, the image processing algorithms may improve accurate object detection, avoid erroneous object detection, and isolate the important object, region, or pattern within an image.
In addition to the image processing techniques described above, the image analysis may include utilizing artificial intelligence (“AI”), such as a machine learning image recognition process, a neural network classification module, any other suitable artificial intelligence (AI) technique, and/or any other suitable image analysis techniques, examples of which will be described in more detail below. Moreover, each of the exemplary image analysis or evaluation processes described below may be used independently, collectively, or interchangeably to extract detailed information regarding the images being analyzed to facilitate performance of one or more methods described herein or to otherwise improve appliance operation. According to exemplary embodiments, any suitable number and combination of image processing, image recognition, or other image analysis techniques may be used to obtain an accurate analysis of the obtained images.
In this regard, the image recognition process may use any suitable artificial intelligence technique, for example, any suitable machine learning technique, or for example, any suitable deep learning technique. According to an exemplary embodiment, the image recognition process may include the implementation of a form of image recognition called region based convolutional neural network (“R-CNN”) image recognition. Generally speaking, R-CNN may include taking an input image and extracting region proposals that include a potential object or region of an image. In this regard, a “region proposal” may be one or more regions in an image that could belong to a particular object or may include adjacent regions that share common pixel characteristics. A convolutional neural network is then used to compute features from the region proposals and the extracted features will then be used to determine a classification for each particular region.
According to still other embodiments, an image segmentation process may be used along with the R-CNN image recognition. In general, image segmentation creates a pixel-based mask for each object in an image and provides a more detailed or granular understanding of the various objects within a given image. In this regard, instead of processing an entire image—i.e., a large collection of pixels, many of which might not contain useful information—image segmentation may involve dividing an image into segments (e.g., into groups of pixels containing similar attributes) that may be analyzed independently or in parallel to obtain a more detailed representation of the object or objects in an image. This may be referred to herein as “mask R-CNN” and the like, as opposed to a regular R-CNN architecture. For example, mask R-CNN may be based on fast R-CNN which is slightly different than R-CNN. For example, R-CNN first applies a convolutional neural network (“CNN”) and then allocates it to zone recommendations on the covn5 property map instead of the initially split into zone recommendations. In addition, according to exemplary embodiments, standard CNN may be used to obtain, identify, or detect any other qualitative or quantitative data related to one or more objects or regions within the one or more images. In addition, a K-means algorithm may be used.
According to exemplary embodiments the image recognition process may further include the implementation of Vision Transformer (ViT) techniques or models. In this regard, ViT is generally intended to refer to the use of a vision model based on the Transformer architecture originally designed and commonly used for natural language processing or other text-based tasks. For example, ViT represents an input image as a sequence of image patches and directly predicts class labels for the image. This process may be similar to the sequence of word embeddings used when applying the Transformer architecture to text. The ViT model and other image recognition models described herein may be trained using any suitable source of image data in any suitable quantity. Notably, ViT techniques have been demonstrated to outperform many state-of-the-art neural network or artificial intelligence image recognition processes.
According to still other embodiments, the image recognition process may use any other suitable neural network process while remaining within the scope of the present subject matter. For example, the step of analyzing the one or more images may include using a deep belief network (“DBN”) image recognition process. A DBN image recognition process may generally include stacking many individual unsupervised networks that use each network's hidden layer as the input for the next layer. According to still other embodiments, the step of analyzing one or more images may include the implementation of a deep neural network (“DNN”) image recognition process, which generally includes the use of a neural network (computing systems inspired by the biological neural networks) with multiple layers between input and output. Other suitable image recognition processes, neural network processes, artificial intelligence analysis techniques, and combinations of the above described or other known methods may be used while remaining within the scope of the present subject matter.
In addition, it should be appreciated that various transfer techniques may be used but use of such techniques is not required. If using transfer techniques learning, a neural network architecture may be pretrained such as VGG16/VGG19/ResNet50 with a public dataset then the last layer may be retrained with an appliance specific dataset. In addition, or alternatively, the image recognition process may include detection of certain conditions based on comparison of initial conditions, may rely on image subtraction techniques, image stacking techniques, image concatenation, etc. For example, the subtracted image may be used to train a neural network with multiple classes for future comparison and image classification.
It should be appreciated that the machine learning image recognition models may be actively trained by the appliance with new images, may be supplied with training data from the manufacturer or from another remote source, or may be trained in any other suitable manner. For example, according to exemplary embodiments, this image recognition process relies at least in part on a neural network trained with a plurality of images of the appliance in different configurations, experiencing different conditions, or being interacted with in different manners. This training data may be stored locally or remotely and may be communicated to a remote server for training other appliances and models. According to exemplary embodiments, it should be appreciated that the machine learning models may include supervised and/or unsupervised models and methods. In this regard, for example, supervised machine learning methods (e.g., such as targeted machine learning) may help identify problems, anomalies, or other occurrences which have been identified and trained into the model. By contrast, unsupervised machine learning methods may be used to detect clusters of potential failures, similarities among data, event patterns, abnormal concentrations of a phenomenon, etc.
It should be appreciated that image processing and machine learning image recognition processes may be used together to facilitate improved image analysis, object detection, color detection, or to extract other useful qualitative or quantitative data or information from the one or more images that may be used to improve the operation or performance of the appliance. Indeed, the methods described herein may use any or all of these techniques interchangeably to improve image analysis process and facilitate improved appliance performance and consumer satisfaction. The image processing algorithms and machine learning image recognition processes described herein are only exemplary and are not intended to limit the scope of the present subject matter in any manner.
Notably, as explained briefly above, not all garments having different colors than the primary load color (e.g., outlier garments) present a risk of color contamination within the load. For example, even if a pair of dark jeans is present within a load of white clothes, color contamination may occur only when the jeans are still bleeding dye. Notably, dyed clothes tend to bleed less color over time, e.g., as they experience more wash cycles within a washing machine appliance. Accordingly, aspects of the present subject matter may generally be directed to systems and methods for recording clothing items that do not present a risk of color contamination in a safelist that may be referenced to determine whether corrective action should be taken. For example, this safelist is identified generally by reference number 190 in
In this regard, for example, step 230 may include determining that the outlier garment is not recorded in the safelist 190. In this regard, image data or other data indicative of a detected garment may be compared to corresponding data stored in the safelist 190. In general, the safelist 190 may include a list of garments that may be washed without a substantial risk for color contamination. For example, the safelist 190 may identify clothes by brand, style, color, or any other data suitable for identifying a particular garment.
In addition, it should be appreciated that the safelist 190 may be stored locally (e.g., on controller 166) or remotely (e.g., on remote server 186). For example, as illustrated in
Step 240 may generally include implementing a responsive action in response to identifying the outlier garment that is not on the safelist. In this regard, for example, if step 220 results in the identification of an outlier garment and step 230 results in a determination that the outlier garment is not on the safelist 190, corrective action may be taken to avoid the performance of a wash cycle which may result in color contamination of the load of clothes 172. For example, according to an exemplary embodiment, implementing the responsive action may include preventing the initiation of a wash cycle with the outlier garment in the load of clothes 172.
Method 200 may further include, at step 250, providing a user notification of the identification of the outlier garment that is not on the safelist. For example, this user notification may be provided via control panel 160, e.g., through communications on display 164. According to still other embodiments, the user notification may be provided directly to the user through a remote device 182 (e.g., such as the user's cell phone) over network 184. According to exemplary embodiments, this user notification may provide a user with an opportunity to verify that the outlier garment is not on the safelist 190, that the outlier garment should be placed on the safelist 190, or to provide an opportunity to manually remove the outlier garment from the load of clothes, or to take any other suitable corrective action. For example, method 200 may include stopping the current operating cycle, operating a drain pump 146 to drain wash tub 124, and/or preventing further operating cycles of washing machine appliance 100 until the user has been notified, the color contamination issue has been addressed, etc.
Notably, method 200 may further include determining that the outlier garment is on the safelist 190. In this regard, image of the outlier garment may be transmitted to remote server 186 and remote server 186 may determine that the outlier garment has in fact been added previously to the safelist 190. Accordingly, in response to determining that the outlier garment is safe to wash within the load of clothes, washing machine appliance 100 may perform the wash cycle in response to determining that the outlier garment is on the safelist 190. Notably, controller 166 of washing machine appliance 100 take further action to ensure that bleeding or color contamination issues do not occur when the outlier garments present. In this regard, for example, performing a wash cycle that includes the outlier garment that is on the safelist may include operating the water supply valve 158 to lower a temperature of the flow of wash fluid during the performance of the wash cycle. In this regard, for example, colder wash fluid tends to cause less dye to bleed from clothes. Other control actions may be taken as well, such as reducing the water level, reducing agitation intensity, etc.
Method 200 may include additional steps for improving the outlier garment detection, interacting with the safelist, or manipulating operation of washing machine appliance if an outlier garment is detected. For example, method 200 may further include steps for entering a particular outlier garment into the safelist 190. In this regard, controller 166 may receive a command to enter the outlier garment into the safelist. For example, this command may come from a user (e.g., via remote device 182) upon receiving notification that the outlier garment is present. In response, controller 166 may prompt the user to position the outlier garment in view of the camera assembly in one or more orientations so that the camera 178 may take images for better garment identification. In this regard, camera assembly 170 may obtain one or more item identification images of the outlier garment and these item identification images may be transmitted to remote server 186 for analysis and storage onto the safelist 190.
In addition, step 220 of identifying the outlier garment may include determining a confidence level associated with the identification of the outlier garment. In the event that the confidence level falls below a predetermined confidence threshold, method 200 may include seeking user confirmation regarding the identification of the outlier garment. In this regard, for example, if method 200 results in a determination that a particular pair of jeans has been identified, but the analysis is not able to identify the jeans with a high confidence, method 200 may further include asking the user to confirm whether the jeans may be included in a particular wash cycle. Notably, this user confirmation or interaction may be performed via remote device 182 (e.g., such that the user's cell phone) or through control panel 160, e.g., via display 164 or an external microphone and speaker.
Notably, when the safelist 190 is updated with additional outlier garments, the algorithms or models for performing the image recognition process may also change. Accordingly, method 200 may include transmitting data indicative of the outlier garment for storage in the safelist on the remote server and receiving an algorithm update for implementing the image recognition process during a subsequent wash cycle. In this regard, the algorithms for performing the image analysis may be stored locally on controller 166 (e.g., as identified generally by reference numeral 192). For example, remote server 186 may include a model or algorithm rebuilding software 192 that redevelops or reprograms software for improved identification relative to items within the safelist 190. Accordingly, when a new item is added to the safelist 190 or when the detection algorithm is otherwise improved, remote server 186 may transmit the new detection algorithm to controller 166 of washing machine appliance 100 for subsequent use.
Referring now briefly to
By contrast, if step 304 results in a determination that the load of clothes has a mixture of dark and light clothes, step 308 may include providing a voice reminder or notification to the user that the load of clothes is mixed. This notification provides user with the opportunity to inspect load to make sure there are no issues, e.g., such as a risk for color contamination. Step 310 may include determining whether to any outlier garments (e.g., dark items) are on the safelist. If the outlier garment is on the safelist, method 300 may proceed to step 306 where the process ends. By contrast, if dark items are detected and are not on the safelist, method 300 may proceed to step 312 where the outlier garment is scanned and to step 314 where the results of the scan are communicated to the user (e.g., by speaker output or communication through remote device 182 or by transmitting an image to remote device 182).
At step 316, method 300 may include determining whether their additional outlier garments that need to be checked. If there are additional outlier garments, the process proceeds again step 310. If there are no additional outlier garments, the process proceeds to step 318 where it is determined whether the new items need to be added to the safelist on the remote server. For example, step 318 may include communicating with the user as to whether the outlier garment has been washed sufficiently to reduce the risk of bleeding dye die in color contamination issues. If the user does not wish to add the item to the safelist, the safelist development process may stop at step 306. By contrast, if a user wishes to add the outlier garment to the safelist, step 320 may include updating the safelist in the cloud with identification data and/or images of the outlier garment. In addition, step 322 may include rebuilding the detection software based on the updated safelist and step 324 may include communicating the updated software to the washing machine appliance for implementation during subsequent wash cycles.
As explained above, aspects of the present subject matter are directed to a cloud based assistant in a connected washer that remembers dark clothes that have stopped releasing dye. This system can notify a user when the dark clothes are mixed with light clothes. In conventional methods, for example, the user may mix the dark clothes and light clothes together, and after washing a new pair of blue jeans may release dye and a light-colored T-shirt may be streaked with blue. On the other hand, some clothes have been used and washed many times, and the user may have confirmed that they have stopped releasing dye and that it is OK to wash them together with light clothes in cold water. But to remember all the clothes that are safe to wash in this manner is difficult. In these scenarios, it may be helpful to have an assistant that reminds the user when the dark clothes are mixed with the light clothes; and which dark clothes are safe to wash with light clothes.
According to exemplary embodiments, the cloud-based system may be used for distinguishing dark clothing from light clothing and may notify the user about mixing of clothes. Then, the user may be allowed to separate the dark clothing from light clothing or use the connected washer to check if any of the dark clothes are in a “safelist” of clothes that have stopped releasing dye. If the user decides to mix light clothes with dark clothes in the safelist, a reminder for using cold water may be sent to the washer. The cloud-based “safelist” for dark clothes that have stopped releasing dye after several washes may be updated by image inputs and voice inputs from connected washers. In addition, the users may update the list from a secured software application. Thus, the cloud-based detection software may be built or rebuilt based on this updated safelist. A “trained assistant” may also be used to recognize all the clothes that have stopped releasing dye. As soon as new clothes are added to the cloud based “safelist”, the detection software may be rebuilt in the cloud, and the new detection software created in the cloud may be deployed to the connected washer through Over-The-Air (OTA) communications. Therefore, the customer can teach the system once, and the system can remember the safelist of clothes forever (unless modified by the user). This can also eliminate the inconvenience of using two loads of washing and at the same time it keeps the risk very low by remembering those safe dark clothes.
This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they include structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.
