Patent Application
20220188551
This application claims the benefit of Taiwan application Serial No. 109143844, filed Dec. 11, 2020, the disclosure of which is incorporated by reference herein in its entirety.
The disclosure relates in general to an activity recognition method based on image and a computer-readable media.
A smart living environment indeed provides convenience and safety to the solitary people. In a smart living environment, activity recognition can be done through the sensors installed at different places of the house. A resident's activity can be recognized according to which sensor/sensors is/are triggered. Since some different activities may trigger similar sensors, the differentiation of different activities becomes difficult. Therefore, how to increase recognition accuracy has become a prominent task for the industries.
According to one embodiment, an activity recognition method applied to an activity recognition system configured to recognize several activities is disclosed. The activity recognition method includes: obtaining an original activity image corresponding to a first time point, wherein the original activity image includes several pixels indicating whether several sensors are triggered; determining an image feature according to a second activity corresponding to a second time point, wherein the second time point is prior to the first time point; integrating the original activity image and the image feature to generate a characteristic activity image; and determining a first activity corresponding to the first time point according to the characteristic activity image.
According to another embodiment, a computer-readable media is disclosed. When the computer-readable media is performed by a processing unit of an activity recognition system configured to recognize several activities, the processing unit is enabled to: obtain an original activity image corresponding to a first time point, wherein the original activity image includes several pixels indicating whether several sensors are triggered; determine an image feature according to a second activity corresponding to a second time point, wherein the second time point is prior to the first time point; integrate the original activity image and the image feature to generate a characteristic activity image, and determine a first activity corresponding to the first time point according to the characteristic activity image.
The above and other aspects of the invention will become better understood with regard to the following detailed description of the preferred but non-limiting embodiment (s). The following description is made with reference to the accompanying drawings.
In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing.
Referring to
The computing module 104 includes a storage unit 1041 and a processing unit 1043. The storage unit 1041 can be any type of fixed or movable memory, such as random access memory (RAM), read-only memory (ROM), flash memory, phase-change memory, hard disk drive (HDD), register, solid-state drive (SSD), other similar elements or a combination of the above elements. The processing unit 1043 can be a central processing unit (CPU), programmable general or special purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), other similar elements or a combination of the above elements.
In an embodiment, at several time points, each of the sensors 102-1˜102-n transmits a signal to the computing module 104 according to whether the sensor is triggered at the time point. At each time point, the computing module 104 records which of sensors 102-1˜102-n are triggered and/or which are not triggered to generate a sensing record corresponding to the time point, wherein two adjacent time points are separated by a sampling interval. The sensing record can be stored in the storage unit 1041 of the computing module 104. The processing unit 1043 of the computing module 104 generates an original activity image according to each sensing record. For example, at a first time point, the computing module 104 obtains a first sensing record through the sensors 102-1˜102-n and generates a first original activity image according to the first sensing record; at a second time point, the computing module 104 obtains a second sensing record through the sensors 102-1˜102-n and generates a second the original activity image according to the second sensing record.
The original activity image can be stored in the storage unit 1041. The original activity image can be a dot matrix including several pixels corresponding to the several sensors 102-1˜102-n. For example, the number of sensors 102-1˜102-n is 50, and the size of the original activity image can be 10 pixels long by 10 pixels wide, wherein 50 of the 100 pixels of the original activity image one-to-one correspond to the sensors 102-1˜102-n, the pixels corresponding to the sensors triggered at the time point can be denoted by a first color (such as black), and the pixels corresponding to the sensors not triggered at the time point can be denoted by a second color (such as white).
The activity recognition system 10 can be configured to recognize several activities, such as “Go Home”, “Go Out”, “Cook”, “Wash Dishes”, “Have Meal”, “Rest”, “Sleep”, “Take Bath”, and “Watch TV”. The number of recognizable activities depends on the activity recognition system 10. In an embodiment, the activity recognition system 10 has a higher recognition ability and can recognize 10 different activities. In another embodiment, the activity recognition system 10 has lower recognition ability and can recognize only 6 different activities.
When several different activities trigger similar sensors, the corresponding activity images may have a large similarity, and activity determination based on activity images may become confused. For example, “Go Out” and “Go Home” are two different activities. When the two different activities trigger similar sensors, “Go Out” may be mis-determined as “Go Home”, or “Go Home” may be mis-determined as “Go Out”. The activity recognition method disclosed in the embodiments of the present invention can effectively avoid the above instances.
Referring to
Firstly, the method begins at step S201, an original activity image corresponding to a first time point is obtained. Detailed descriptions of the original activity image can be obtained with reference to the above disclosure. In an embodiment, the processing unit 1043 can access the storage unit 1041 to obtain the original activity image.
Next, the method proceeds to step S203, an image feature is determined according to a second activity corresponding to a second time point, wherein the second time point is prior to the first time point. For example, the second time point is prior to the first time point by a sampling interval. In an embodiment, the first time point and the second time point are sampling time points. Suppose the sampling interval is 1 second, then the second time point is 1 second before the first time point. That is, the second activity is an activity corresponding to the second time point recognized by the processing unit 1043 using the present method. The image feature may include one or more pixels, and can be any form, such as text, pattern, or color. The number of image features can be identical to the number of activities recognizable to the activity recognition system 10. For example, the activity recognition system 10 can recognize 10 different activities one-to-one corresponding to 10 different image features. That is, each activity corresponds to a unique image feature. In an embodiment, two different activities will not correspond to the same mage feature. In an embodiment, step S203 is performed before step S201.
Then, the method proceeds to step S205, the original activity image and the image feature are integrated to generate a characteristic activity image. In an embodiment, the processing unit 1043 replaces a part of the original activity image with an image feature to generate a characteristic activity image, that is, the size of the characteristic activity image is equivalent to that of the original activity image. In an embodiment, the processing unit 1043 attaches (or connects) an image feature to the original activity image to generate a characteristic activity image, that is, the size of the characteristic activity image is greater than the original activity image. For the present step to be better understood, the integration of the original activity image and the image feature is described with several practical examples below. In an embodiment, the image feature is a 2×2 square with a grayscale value (4 pixels). During the integration process, the processing unit 1043 replaces the pixels of the original activity image not used to represent the sensors, such as one of the four corners, with an image feature to generate a characteristic activity image. In another embodiment, the image feature is a column of pixels with a grayscale value. During the integration process, the processing unit 1043 connects the image feature to the topmost or bottommost column of pixels of the original activity image to generate a characteristic activity image. In an alternate embodiment, the image feature is formed of several pixels with an RGB value. During the integration process, the processing unit 1043 uses the image feature as an outer frame of the original activity image and attaches the image feature to the outer edge of the original activity image to generate a characteristic activity image. The above examples are for explanatory purposes only, not for limiting the scope of the present invention.
Referring to
Then, the method proceeds to step S207, a first activity corresponding to the first time point is determined according to the characteristic activity image. In an embodiment, the computer-readable media includes is used to implement one or more procedures, such as neural network procedures. The neural network can be used to determine the first activity corresponding to the first time point according to the characteristic activity image.
Through the above method, the image feature representing the second activity determined at the second time point prior to the first time point can be “added” to the activity image obtained at the first time point to generate a characteristic activity image. Thus, the characteristic activity image contains the image feature representing the previous activity, and the accuracy of determining the current activity will be increased if the previous activity can be taken into consideration. Particularly, when the current activity is one of two different activities such as “Go Home” and “Go Out” that trigger similar sensors. As the previous activity is taken into consideration, the probability of mis-determination can be effectively reduced.
Firstly, the method begins at step S301, each activity recognizable to the activity recognition system is paired with other activities by the activity recognition system to form several activity pairs, and a similarity between two activities corresponding to each activity pair is calculated one by one. In an embodiment, the similarity between two activities refers to a parameter generated by quantifying the overlapping situation between the sensor triggered by one of the two activities and the sensor triggered by the other one of the two activities. That is, similarity represents the degree of overlapping between the sensors triggered by two activities, and the larger the degree of overlapping, the larger the similarity. In an embodiment, the similarity between two activities is calculated using cosine similarity. In another embodiment, any mathematical tools that can be used to quantify the degree of similarity between the sensors triggered by two activities are used to calculate similarity.
For example, suppose the activity recognition system can recognize 6 activities, namely “Go Home”, “Go Out”, “Cook”, “Wash Dishes”, “Rest”, and “Sleep”. The activity pairs include “Go Home” pairing with other 5 activities, “Go Out” pairing with other 4 activities (“Go Home” pairing with “Go Out” is not repeated), “Cook” pairing with other 3 activities, and the rest activity pairs can be obtained by the same analogy. Thus, the activity pairs include [“Go Home”-“Go Out], [“Go Home”-“Cook”], [“Go Home”-“Wash Dishes”], [“Go Home”-“Rest”], [“Go Home”-“Sleep”], [“Go Out”-“Cook”], [“Go Out”-“Wash Dishes”], and the rest activity pairs can be obtained by the same analogy. When performing step S301, the similarity between two activities of each activity pair is calculated. In the above example, the similarity between “Go Home” and “Go Out”, the similarity between “Go Home” and “Cook”, the similarity between “Go Home” and “Wash Dishes”, the similarity between “Go Home” and “Rest”, the similarity between “Go Home” and “Sleep”, the similarity between “Go Out” and “Cook”, the similarity between “Go Out” and “Wash Dishes”, the similarity between “Go Out” and “Rest”, and the similarity between “Go Out” and “Sleep” are calculated, and the rest similarities can be obtained by the same analogy. The similarities can be represented by the similarity percentage in the form of Table 1 below:
Next, the method proceeds to step S303, an arrangement sequence of activity pairs is determined according to the similarity corresponding to each activity pair. In an embodiment, the arrangement sequence of activity pairs is determined according to the similarities sorted in descending order. In the above example, the similarities sorted in descending order are 95%, 87%, 43%, 37%, . . . , 7% and 6%. Thus, the arrangement sequence of activity pairs is: [“Go Home”-“Go Out”], [“Cook”-“Wash Dishes”], [“Go Home”-“Rest”], [“Rest”-“Sleep”], . . . , [“Sleep”-“Wash Dishes”], [“Rest”-“Wash Dishes”].
Then, the method proceeds to step S305, each activity is allocated to a unique image feature according to the arrangement sequence and a probability distribution of the previous activity of each activity. In an embodiment, the previous activity of each activity refers to the activity which occurred prior to the current activity by a sampling interval (that is, the activity which occurred at the previous time point). In an embodiment, the probability distribution of the previous activity of each activity can be obtained through observation and counting. For example, the activity “Go Home” is observed 1000 times and the frequency of the previous activity of the activity “Go Out” among the 1000 observations is counted. Suppose of the 1000 observations, the previous activity is observed as “Go Out” for 700 times and is observed as “Go Home” for 300 times. Then, the probability distribution of the previous activity of the activity “Go Home” is as follows: “Go Out” has a probability of 70%, and “Go Home” has a probability of 30%. The activity “Go Out” 1000 is observed 1000 times, and the frequency of the previous activity of the activity “Go Out” among the 1000 observations is counted. Suppose of the 1000 observations, the previous activity is observed as “Rest” for 540 times, is observed as “Wash Dishes” for 310 times, and is observed as “Go Home” for 150 times. Then, the probability distribution of the previous activity of the activity “Go Out” is as follows: “Rest” has a probability of 54%, “Wash Dishes” has a probability of 31%, and “Go Home” has a probability of 15%, and the probabilities of rest activities can be obtained by the same analogy. In an embodiment, when arranging the image features for activity pairs, the higher the rank in the arrangement sequence the higher the priority. Let the previous example be taken for example. In the arrangement sequence, the activity pair ranked first is [“Go Home”-“Go Out”] with a similarity of 95%, the activity pair ranked second is [“Cook”-“Wash Dishes”] with a similarity of 87%, and the activity pair ranked third is [“Go Home”-“Rest”] with a similarity of 43%. Therefore, when arranging the image features, the activity pair [“Go Home”-“Go Out”] has the first priority, the activity pair [“Cook”-Wash Dishes”] has the second priority, the activity pair [“Go Home”-“Rest”] has the third, priority, and the rest can be obtained by the same analogy. Details of step S305 can be obtained with reference to the flowchart of
In step S401, whether one activity of the activity pair with the largest probability among the probability distribution of the previous activity already has a corresponding image feature is determined. If yes, the method proceeds to S403; otherwise, the method proceeds to S405.
In step S403, whether the other activity of the activity pair with the largest probability among the probability distribution of the previous activity has a corresponding image feature is determined. If yes, the method proceeds to S407; otherwise, the method proceeds to S405.
In step S405, the activity is allocated to an unmatched candidate image feature of several candidate image features and used as an image feature corresponding to the activity.
In step S407, whether each activity has a corresponding image feature is determined. If yes, the process terminates; otherwise, the process proceeds to step S409.
In step S409, whether the arrangement sequence has reached the end is determined. If yes, the process proceeds to S413; otherwise, the method proceeds to S411.
In step S411, the activity pair ranked second in the arrangement sequence is considered, and the process returns to S401.
In step S413, each unmatched activity is allocated to an unmatched candidate image feature of the image features.
The flowchart of
Thus, after the original activity image is obtained, 1 row (or 1 column) of pixels with the grayscale value representing the second activity (that is, the activity corresponding to the previous time point) is “added to” the original activity image to generate a characteristic activity image, such that the neural network can more accurately determine the current activity according to the added image feature (that is, the previous activity).
In an embodiment, a step S304 is provided between step S303 and step S305. In step S304, the arrangement sequence is adjusted according to an occurrence frequency of each activity. For a specific activity, the occurrence frequency represents the number of times for which the specific activity is performed by a user over several time points. For example, for the activity “Sleep”, the number of times for which the activity “Sleep” is performed by the user can be obtained by observing 1000 time points, and the rest activities can be obtained by the same analogy. Thus, the occurrence frequency of each activity can be obtained by observing the number of occurrences of each activity among the 1000 observations. For an activity with an occurrence frequency lower than a specific threshold (the specific threshold can be set according to actual needs), the activity may be mis-determined because the occurrence frequency of the activity is too low (the activity has a lower probability). Therefore, in step S304, the rank of one or more activity pairs containing the activity with an occurrence frequency lower than the specific threshold is adjusted forward. For example, the rank is adjusted to be higher than the activity pair with the largest similarity. In step S405, the image feature is allocated according to the adjusted arrangement sequence. In the above example, suppose the occurrence frequency of the activity “Sleep” is lower than the specific threshold. When performing step S304, the rank of the activity pair [“Sleep”-“Rest”], which has the largest similarity among all activity pairs containing the activity “Sleep”, is adjusted to be higher than the activity pair [“Go Home”-“Go Out”].
In an embodiment, in step S403, the activity pairs can be divided into several problem groups according to several similarity thresholds, then the arrangement sequence is decided according to the problem groups and the similarity thresholds. For example, 3 similarity thresholds are set as 70%, 50%, 30%, and 4 problem groups are set as a serious problem group, secondary problem group, ordinary problem group, and no problem group. In the present example, the activity pairs with a similarity between 71%˜100% are allocated to the serious problem group; the activity pairs with a similarity between 51%˜70% are allocated to the secondary problem group, and the rest activity pairs can be obtained by the same analogy. When deciding the arrangement sequence, the activity pairs in the serious problem group are ranked first, and the activity pairs in the secondary problem group are ranked second, and the activity pairs in the rest groups can be obtained by the same analogy.
In an embodiment, in step S403, the arrangement sequence can be adjusted according to the occurrence frequency and the similarity thresholds. For example, 4 problem groups are set as a serious problem group, primary problem group, secondary problem group, and no problem group, and 2 similarity thresholds are set as 70% and 50%. The activity pairs with an occurrence frequency lower than a specific threshold are allocated to the serious problem group; the activity pairs with a similarity between 71%˜100% are allocated to the primary problem group, the activity pairs with a similarity between 51%˜70% are allocated to the secondary problem group; and the rest activity pairs can be obtained by the same analogy. When deciding the arrangement sequence, the activity pairs in the serious problem group are ranked first, the activity pairs in the primary problem group, the secondary problem group, and the no problem group are ranked subsequently.
It should be noted that the order of steps as indicated in
In the above embodiments, for each activity pair, the allocation of image features considers the activity with the largest probability among the previous activities of the activity pair. In an alternate embodiment, for each activity pair, the allocation of image features can consider n activities with n largest probabilities among the previous activities of the activity pair, wherein n is an integer greater than 1. For example, in an embodiment, the activity pair ranked first in the arrangement sequence is considered, and the two activities with two largest probabilities among the previous activities of each of the two activities of the activity pair are found, and an image feature is allocated to the 4 activities respectively. Then, the activity pair ranked second in the arrangement sequence is considered, and the rest activity pairs can be obtained by the same analogy.
Through an embodiment of the present invention, the probability of mis-determining different activities as similar activities when similar sensors are triggered by different activities can be effectively reduced, such that the accuracy and safety of the activity recognition system can be increased.
It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 109143844 | Dec 2020 | TW | national |
