Patent Application
20220218169
The present disclosure relates to a cleaner system including a cleaner that can autonomously travel to clean a home and a common area such as a corridor of a public facility or an office building, and an information processing apparatus, to a cleaner used in the cleaner system, and to a dirt determination program.
An autonomous traveling cleaner known in the art acquires a thermal image or measures a concentration of gas to detect excrement of a living being on a floor surface, and switches an operation mode (see, for example, JP 2018-515191 A (hereinafter, referred to as “Patent Document 1”)).
On the other hand, dirt adhering to a floor surface including a carpet, such as dust accumulating on the floor surface and sand brought to the floor surface from shoes, is difficult to be detected due to a design or the like on the floor surface under certain conditions.
The present disclosure provides a cleaner system, a cleaner, and a dirt determination program capable of appropriately determining dirt on a floor surface.
A cleaner system from one aspect of the present disclosure includes an autonomous traveling cleaner that autonomously travels to clean a floor surface, and an information processing apparatus that processes information about dirt on the floor surface. The information processing apparatus includes a first information generator that generates first information about the dirt based on a plurality of pieces of image data indicating the floor surface. The cleaner includes an imaging device that images the floor surface. The cleaner system includes a second information generator that generates second information based on image data acquired from the imaging device, a determination unit that determines whether the second information indicates dirt, based on the first information and generates a determination result, and a cleaning controller that changes a cleaning operation on the floor surface on which the second information is acquired, based on the determination result of the determination unit.
Further, a cleaner from another aspect of the present disclosure includes an imaging device that images a floor surface, an information acquisition unit that acquires first information about dirt based on a plurality of pieces of image data indicating the floor surface and a determination model trained by the first information, a second information generator that generates second information based on image data acquired from the imaging device, a determination unit that determines whether the second information indicates the dirt, based on the first information or using the determination model trained by the first information and generates a determination result, and a cleaning controller that changes a cleaning operation on the floor surface on which the second information is acquired, based on the determination result of the determination unit.
Further, a dirt determination program from another aspect of the present disclosure is used in a cleaner system that includes an autonomous traveling cleaner that autonomously travels to clean a floor surface and an information processing apparatus that processes information about dirt on the floor surface. The dirt determination program is executed by a processor to achieve a first information generator that generates first information about dirt based on a plurality of pieces of image data indicating the floor surface, a second information generator that generates second information based on image data indicating the floor surface, a determination unit that determines whether the second information indicates dirt, based on the first information and generates a determination result, and a cleaning controller that changes a cleaning operation on the floor surface on which the second information is acquired, based on the determination result of the determination unit.
The present disclosure can provide a cleaner system, a cleaner, and a dirt determination program capable of reducing erroneous determination of dirt on a floor surface and appropriately cleaning the floor surface.
An exemplary embodiment of a cleaner system, a cleaner, and a dirt determination program according to the present disclosure will be described below with reference to the drawings. Note that the following exemplary embodiment is merely an example for describing the present disclosure, and does not limit the present disclosure. For example, a shape, a structure, a material, a component, a relative positional relationship, a connection state, a numerical value, a mathematical expression, description of each step in a method, an order of each step, and the like described in the following exemplary embodiment are merely examples, and the present disclosure may include matters that are not described below. Further, geometric expressions such as “parallel” and “orthogonal” may be used, but these expressions do not indicate mathematical strictness, and include substantially acceptable errors, deviations, and the like. In addition, expressions such as “simultaneous” and “identical” include substantially acceptable ranges.
Further, the drawings are schematic drawings in which emphasis, omission, or ratio adjustment is appropriately performed in order to describe the present disclosure. Thus, actual shapes, positional relationships, and ratios are different from those of the drawings.
Further, a plurality of inventions may be comprehensively described below as one exemplary embodiment. Further, some parts of the content are described below as any components regarding the present disclosure.
Information processing apparatus 110 is an apparatus that processes information about dirt on a floor surface and provides the processed information to cleaner 130, and can transmit and receive information to and from cleaner 130 via a network. In the present exemplary embodiment, information processing apparatus 110 is a so-called server including a processor, a storage device, various interfaces, and the like. Information processing apparatus 110 includes data acquisition unit 111 and first information generator 112 as processing units achieved by causing the processor to execute a program. In the present exemplary embodiment, information processing apparatus 110 includes model training unit 113 and first floor type classifier 114.
Data acquisition unit 111 is a processing unit that acquires image data indicating a floor surface. Data acquisition unit 111 acquires the image data not from limited devices, and can acquire from a mobile terminal such as a smartphone or a tablet terminal, cleaner 130, a digital camera, or the like via a network or directly.
First information generator 112 generates information indicating a floor surface and a state of dirt on the floor surface based on the image data indicating the floor surface. A specific example of the first information is a feature amount obtained by digitalizing an arrangement (design or pattern) or the like in at least one item in the image data such as hue, saturation, and lightness. First information generator 112 generates the first information as the feature amount from the image data using any method, and may generate the first information using artificial intelligence. First information generator 112 may exclude, from the acquired image data, an obstacle on the floor surface analyzed as being different from the floor surface by an image analysis or the like. This enables generation of highly reliable first information.
First floor type classifier 114 is a processing unit that classifies a type of the floor surface indicated by the image data acquired by data acquisition unit 111. First floor type classifier 114 detects a floor type such as a wood floor, tiles, a thin-piled carpet, a thick-piled carpet, or tatami mats, based on the image data to classify the types, and associates the image data with data indicating the floor types. Note that first floor type classifier 114 may classify the floor types based on a hue, saturation, lightness, a design, and the like of the floor surface. Further, floor types may be classified based on an input from a user.
Model training unit 113 acquires a plurality of different pieces of first information as feature amounts, and trains a determination model using the first information, a human evaluation and another evaluation of a state of dirt on the floor surface indicated by the image data. The determination model is a nonlinear regression model used for deep learning, machine learning, and the like.
In the present exemplary embodiment, first floor type classifier 114 associates the first information generated from the image data with the floor types, and thus model training unit 113 trains a plurality of determination models trained for each classified floor. The trained determination model is provided to cleaner 130 via a network or the like.
According to the present exemplary embodiment, cleaner 130 includes body 131 on which various components are mounted, traveling unit 132 that moves body 131, cleaning unit 133 that collects dust present on a floor surface, imaging device 137, controller 135, position sensor 136, and obstacle sensor 145.
Body 131 is a housing that houses traveling unit 132, controller 135, and the like. Its upper portion is detachable from its lower portion. Bumper 139 which is displaceable with respect to body 131 is attached to an outer peripheral portion of body 131. Further, as illustrated in
Traveling unit 132 is a device that causes cleaner 130 to travel based on an instruction from controller 135. In the present exemplary embodiment, cleaner 130 includes position sensor 136, and traveling unit 132 also functions as a device that moves position sensor 136. Traveling unit 132 includes wheels 140 that move along a floor and a traveling motor (not illustrated) that applies a torque to wheels 140. Caster 142 is mounted as an auxiliary wheel on a bottom surface of body 131. Independently controlling the rotation of two wheels 140 enables cleaner 130 to freely travel forward, travel backward, turn left, turn right, and the like.
Cleaning unit 133 is a unit that sucks dust through suction port 138 into body 131 and holds the dust. Cleaning unit 133 includes an electric fan (not illustrated) and dust holding unit 143. The electric fan sucks air inside dust holding unit 143 and discharges the air to the outside of body 131 to suck dust through suction port 138 and store the dust into dust holding unit 143. Cleaning unit 133 includes main brush 141 that sweeps up dust and the like on the floor surface into suction port 138, and side brush 134 that sweeps and collects dust for sucking the dust through suction port 138.
Cleaning unit 133 includes cleaning member 146. Cleaning member 146 is not particularly limited, but, in the present exemplary embodiment, may have a replaceable sheet that can be pressed against the floor surface to wipe a floor surface. Note that cleaning unit 133 may include a detergent application device that sprays detergent onto a floor surface and an infiltration device that infiltrates detergent into cleaning member 146.
Position sensor 136 detects a positional relationship including a distance between cleaner 130 and an object including a wall or the like existing around cleaner 130 on the floor surface and a direction of the object. Further, position sensor 136 also can get a self-position of cleaner 130 from the information about the direction and distance detected by position sensor 136. The type of position sensor 136 is not particularly limited, and its examples are a light detection and ranging (LiDAR) camera and a time of flight (ToF) camera that emit light and detect a position and a distance based on returned light reflected by an obstacle. A further example of position sensor 136 is a compound eye camera that acquires illumination light or natural light reflected by an obstacle as an image and acquires the position and the distance based on parallax.
Obstacle sensor 145 is a sensor for detecting an obstacle placed on a floor surface. In the present exemplary embodiment, obstacle sensor 145 is a sensor capable of detecting an obstacle that is not so high above a floor surface, such as paper, cloth, or excrement of a living being placed on the floor surface.
Note that cleaner 130 may include a sensor in addition to position sensor 136 and obstacle sensor 145. Cleaner 130 may include, for example, a floor surface sensor that is disposed at a plurality of places on the bottom surface of body 131 and detects whether a descending step such as a staircase is present. Further, cleaner 130 may include an encoder that is provided in traveling unit 132 and detects a rotation angle of each of the pair of wheels 140 rotated by the traveling motor. Further, cleaner 130 may include an acceleration sensor that detects acceleration during traveling of cleaner 130, and an angular velocity sensor that detects an angular velocity during turning of cleaner 130. Cleaner 130 may include a contact sensor that detects displacement of bumper 139 to detect collision with an obstacle.
Imaging device 137 is a device that images a floor surface. The type of imaging device 137 is not particularly limited, and in the present exemplary embodiment, cleaner 130 includes a digital camera having an optical system and an imaging element as imaging device 137.
Cleaner 130 includes second information generator 171, determination unit 172, and cleaning controller 173 as processing units achieved by causing a processor included in controller 135 to execute a program. In the present exemplary embodiment, cleaner 130 further includes notification unit 174, obstacle detector 175, travel controller 176, comparative information acquisition unit 177, and second floor type classifier 178.
Second information generator 171 generates second information based on image data acquired by imaging device 137. In the present exemplary embodiment, second information generator 171 uses a method similar to the method with which first information generator 112 of information processing apparatus 110 generates the first information, and generates a feature amount as the second information from the image data. The second information is information indicating the floor surface indicated in the image acquired by imaging device 137 and the state of dirt on the floor surface. Second information generator 171 may generate the second information from image data excluding the obstacle detected by obstacle detector 175. As a result, the data amount of the image data for generating the second information can be reduced, and the second information can be acquired with high accuracy and at high speed.
Note that second information generator 171 may associate the self-position of cleaner 130 obtained by position sensor 136 with the second information generated based on the image data.
Second floor type classifier 178 is a processing unit that classifies the type of floor surface indicated by the image data imaged by imaging device 137. Second floor type classifier 178 detects a floor type such as a wood floor, tiles, a thin-piled carpet, a thick-piled carpet, or tatami mats similar to those detected and classified by first floor type classifier 114, based on the image data to classify the floor type, and associates the second information with data indicating the floor type.
A classification method used by second floor type classifier 178 is not particularly limited, and for example, the floor type may be classified by analyzing the image data acquired by imaging device 137. Further, the floor type may be classified based on position information acquired by position sensor 136 during the imaging of a floor surface. Further, the floor type may be classified based on an input from a user.
Determination unit 172 determines whether the second information generated by second information generator 171 indicates dirt, based on the first information acquired from information processing apparatus 110, and generates a determination result. In the present exemplary embodiment, comparative information acquisition unit 177 acquires a determination model trained by the first information in information processing apparatus 110, and determination unit 172 uses the acquired determination model to determine whether the second information indicates dirt and generate a determination result.
In the present exemplary embodiment, model training unit 113 of information processing apparatus 110 trains a plurality of determination models corresponding to a plurality floor types, and comparative information acquisition unit 177 acquires the plurality of types of trained models.
Determination unit 172 inputs the second information into the determination model corresponding to the second information classified by second floor type classifier 178, and generates a determination result indicating the type of dirt, the degree of dirt, and the like indicated by the second information. Determination accuracy can be improved by making the determination of the dirt using the determination model corresponding to a floor type.
Note that determination unit 172 may compare the plurality of pieces of first information with the generated second information without using the trained model to determine the degree of the dirt based on the first information similar to the second information, and generate a determination result.
Cleaning controller 173 changes a cleaning operation on the floor surface, on which the second information is acquired, in accordance with the determination result of determination unit 172. In the present exemplary embodiment, cleaning controller 173 causes cleaning unit 133 to perform cleaning corresponding to the self-position, roughly based on a cleaning plan, but executes changes in a suction force, whether the brush is rotated, whether cleaning member 146 is used, and the like in accordance with the determination result of determination unit 172. For example, when determination unit 172 determines that the state of dirt on the floor surface is dust adhered onto a thin-piled carpet and generated in daily life, cleaning controller 173 causes cleaning unit 133 to operate main brush 141 and side brush 134 and sweep and collect the dust to suck the dust through suction port 138. Further, when determination unit 172 determines that the state of dirt on the floor surface is a liquid on a wood floor, cleaning controller 173 stops the suction from main brush 141 and side brush 134 through suction port 138. Cleaning controller 173 then causes cleaning unit 133 and traveling unit 132 to move and push cleaning member 146 against the floor surface and to move cleaning member 146 along the liquid. Note that when determination unit 172 determines that the dirt cannot be handled by cleaner 130, additionally based on the information from obstacle sensor 145, cleaning controller 173 may regard the dirt as an obstacle and cause traveling unit 132 to avoid the obstacle.
Notification unit 174 notifies of the determination result in determination unit 172. Examples of the notification are the state of dirt on a floor surface, the type of dirt, and presence of dirt that cannot be handled by cleaner 130 on a floor surface. A notification method used by notification unit 174 is not particularly limited, and examples thereof include displaying an image including a text on display unit 103, outputting a sound including a melody and a warning sound, and the like.
Operations of cleaner system 100 will be described below.
First, a flow of an operation for generating the first information will be described.
Cleaner 130 is then caused to travel in accordance with a preset cleaning plan, and the floor surface is imaged at predetermined intervals by imaging device 137 (S102). The imaged image data is associated with the self-position information about cleaner 130 acquired by position sensor 136, the information about the floor type acquired by cleaner 130 thorough the sensor or a user's input, and the like, and the associated information is transmitted to information processing apparatus 110 (S103). Note that cleaner 130 may remove an obstacle from the image data based on the information of obstacle sensor 145 and output the image data.
First information generator 112 of information processing apparatus 110 generates first information that is a feature amount, based on the acquired image data (S104). Model training unit 113 trains a determination model based on the generated first information and information that is image data acquired by imaging a relatively clean floor surface (S105). The trained determination model is registered in the storage device included in information processing apparatus 110 (S106).
Note that the determination model may be registered after being associated with the self-position information indicating the imaging position acquired from the cleaner, the information about the type of the imaged floor, and the like. Further, although the case where the determination model is registered has been exemplified, the first information may be registered in the storage device without using the determination model. The self-position information, the floor information, and the like may be associated with the first information. In addition, information processing apparatus 110 may acquire image data from a plurality of cleaners 130, a mobile terminal such as a smartphone, or the like to generate first information, and train the determination model.
A flow of a cleaning operation performed by cleaner 130 will be described below.
Cleaner 130 is caused to travel in accordance with a preset cleaning plan and clean the floor surface while imaging device 137 is imaging the floor surface on the front side in a traveling direction of cleaner 130 at predetermined intervals (S202). Second information generator 171 generates the second information based on the imaged image data (S203).
Determination unit 172 generates a determination result that is generated by comparing the first information with the second information or inputting the second information into the determination model and indicates whether the second information indicates dirt (S204). Note that when position information, information indicating the floor type, and the like are associated with the first information or determination model acquired by comparative information acquisition unit 177, determination unit 172 selects the first information or determination model associated with the information corresponding to the floor surface type classified by second floor type classifier 178, and generates a determination result.
Cleaning controller 173 changes the cleaning operation on the floor surface, on which the second information is acquired, in accordance with the determination result of determination unit 172 (S205). Note that the cleaning operation is not changed depending on the determination result. Cleaner 130 repeats the operations from the floor surface imaging (S202) to the change of the cleaning operation (S205) until the cleaning is completed (S206).
In cleaner system 100 according to the present exemplary embodiment, an actual state of a floor surface is determined from second information generated based on an image acquired by imaging device 137 included in cleaner 130, based on accumulated first information or a determination model trained by the first information, and the cleaning operation can be changed in accordance with a determination result. This can reduce erroneous detection of dirt due to a design of the floor or the like and appropriately clean the floor surface.
Further, dirt detection accuracy can be improved by changing data for determination in accordance with a floor type.
Further, recognition of an obstacle can improve the determination accuracy of dirt on a floor surface, and thus the cleaning operation can be changed to an appropriate operation including an avoiding operation in accordance with the type of the obstacle.
Note that the present disclosure is not limited to the above exemplary embodiment. For example, in the present disclosure, another exemplary embodiment may be achieved by combining any components described in the specification or excluding some of the components. Further, the present disclosure also includes modifications conceivable by those skilled in the art and obtained by variously modifying the above-described exemplary embodiment without departing from the spirit of the present disclosure, that is, the meaning indicated by the words described in the claims.
For example, the image data for generating the first information has been described as the image data regarding specific regions acquired by imaging the floor surface to be cleaned by cleaner 130, but the image data may be so-called big data acquired by imaging unspecified regions at a plurality of locations.
Further, a cleaning preparation mode may be executed before a shift to the cleaning mode. Specifically, second information at a plurality of locations may be generated by causing cleaner 130 to travel in accordance with the cleaning plan or along any route without performing the cleaning operation and to perform imaging more than once. Then, the cleaning plan in accordance with dirt on a floor surface may be updated or generated based on the second information and the first information.
Although cleaner 130 and information processing apparatus 110 have been described as separate components, cleaner 130 may include information processing apparatus 110.
Further, at least one of second information generator 171, second floor type classifier 178, and determination unit 172 described as being included in cleaner 130 may be achieved in information processing apparatus 110.
Further, at least one of data acquisition unit 111, first information generator 112, first floor type classifier 114, and model training unit 113 described as being included in information processing apparatus 110 may be achieved in cleaner 130.
Further, in the above description, the first information, the second information, and the determination model are classified for each floor type, and the determination is made for each floor type. However, dirt on a floor may be determined using the first information or one determination model trained by including the floor type as a parameter in the second information or by the first information including the floor type.
The present disclosure is usable in a cleaner system including an autonomous travel type cleaner.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-002186 | Jan 2021 | JP | national |
