Patent Application
20230206640
This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2021-215309, filed on Dec. 28, 2021, the entire contents of which are incorporated herein by reference.
The embodiments discussed herein are related to a computer-readable recording medium, an information processing method, and an information processing apparatus.
There is a known behavior recognition technology for recognizing behaviors of persons from video image data. For example, there is a known technology for recognizing, from video image data captured by cameras or the like, motions or behaviors performed by persons by using skeleton information on the persons included in the video image data. In recent years, with the spread of self-service checkout counters in supermarkets or convenience stores or the spread of monitoring cameras in schools, trains, public facilities, or the like, an introduction of human behavior recognition is in progress.
According to an aspect of an embodiment, a non-transitory computer-readable recording medium stores therein an information processing program that causes a computer to execute a process. The process includes acquiring video image data that includes target objects including a person and an object, first specifying, by inputting the acquired video image data to a first machine learning model, a relationship between each of the target objects included in the acquired video image data, second specifying, by using a feature value of the person included in the acquired video image data, a behavior of the person included in the video image data, and predicting, by inputting the specified behavior of the person and the specified relationship to a probability model, a future behavior or a future state of the person.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
However, a behavior of a person recognized by the behavior recognition technology described above indicates the behavior that is currently performed or that was performed in the past by the person. Therefore, in some cases, even if a countermeasure is taken after recognition of a predetermined behavior performed by the person, it may be too late to take the countermeasure.
Preferred embodiments will be explained with reference to accompanying drawings. Furthermore, the present invention is not limited to the embodiments. In addition, each of the embodiments can be used in any appropriate combination as long as they do not conflict with each other.
Overall Configuration
Each of the plurality of cameras 2 is one example of a monitoring camera that captures an image of a predetermined region in the store 1, and transmits data of the captured video image to the information processing apparatus 10. In a description below, the data of the video image is sometimes referred to as “video image data”. Furthermore, the video image data includes a plurality of frames arranged in time series. A frame number is assigned to each of the frames in an ascending order in time series. A single frame is image data of a still image that is captured by one of the cameras 2 at a certain timing.
The information processing apparatus 10 is one example of a computer that analyzes each of the pieces of image data captured by the respective plurality of cameras 2. Furthermore, each of the plurality of cameras 2 and the information processing apparatus 10 are connected with each other by using various networks, such as the Internet or a dedicated line, irrespective of a wired or wireless manner.
In recent years, monitoring cameras are installed not only in the store 1, but also in town, station platforms, and the like, and various services are provided to realize a safe and secure society by using video image data acquired by the monitoring cameras. For example, services for detecting an occurrence of shoplifting, an occurrence of an accident, an occurrence of a suicide by jumping, or the like, and making good use of the detection for dealing with the aftermath are provided. However, all of the services that are currently provided cope with post-detection, and, from the viewpoint of prevention beforehand, it is hard to say that the video image data is effectively used for a sign of shoplifting, a possibility of a suspicious person, a sign of an attack of illness, a sign of a dementia, Alzheimer' disease, or the like that is difficult to be determined at a glance.
Thus, in the first embodiment, the information processing apparatus 10 that implements “behavior prediction” to predict a future behavior or a future internal state of a person by combining “behavior analysis” for analyzing a current facial expression or a current behavior of the person and “context sensing” for detecting a surrounding environment, an object, and a relationship with the environment and the object will be described.
Specifically, the information processing apparatus 10 acquires video image data that includes target objects including a person and an object. Then, the information processing apparatus 10 specifies a relationship between each of the target objects included in the video image data by using a relationship model for specifying the relationship between the target objects included in the video image data. In contrast, the information processing apparatus 10 specifies a current behavior of the person included in the video image data by using a feature value of the person included in the video image data. After that, the information processing apparatus 10 predicts, by inputting the specified current behavior of the person and the specified relationship to a probability model, a future behavior of the person, such as a sign of shoplifting, or a future state of the person, such as Alzheimer.
For example, as illustrated in
Furthermore, the information processing apparatus 10 performs behavior recognition on a current behavior of the person by using both of a behavior analyzer and a facial expression analyzer. Specifically, the behavior analyzer acquires, by inputting the video image data to a trained skeleton recognition model, skeleton information that is related to the person and that is one example of a feature value. The facial expression recognizer acquires, by inputting the video image data to a trained facial expression recognition model, facial expression information that is related to the person and that is one example of a feature value. Then, the information processing apparatus 10 refers to a behavior specific rule that is defined in advance, and recognizes a current behavior of the person corresponding to the specified combination of the skeleton information and the facial expression information on the person.
After that, the information processing apparatus 10 inputs the relationship between the person and another person or the relationship between the person and the object and the current behavior of the person to the probability model that is one example of a model constituted by using a hidden Markov model or the like, and acquires a result of the behavior prediction of the future behavior of the person.
Here, regarding the behavior that is predicted by the information processing apparatus 10, it is possible to perform various predictions from a short term prediction to a long term prediction.
Specifically, the information processing apparatus 10 predicts, as a super short term prediction to be performed after a few seconds to a few minutes, an occurrence or a need of a “human support by a robot”, an “online communication support”, or the like. The information processing apparatus 10 predicts, as a short term prediction to be performed after a few hours, an occurrence of an unexpected event, such as a “purchase behavior in a store”, a “crime, such as shoplifting or stalking”, or a “suicide behavior”, or an event that occurs with a small amount of movement from a place in which a current behavior is performed. The information processing apparatus 10 predicts, as a medium term prediction to be performed after a few days, an occurrence of a planned crime, such as a “police box attack” or “domestic violence”. The information processing apparatus 10 predicts, as a long term prediction to be performed after a few months, an occurrence of a potential event (state), such as an “improvement in grade of study or sales” or a “prediction of disease, such as Alzheimer” that is not recognizable by an appearance.
In this way, the information processing apparatus 10 is able to detect a situation in which a countermeasure is needed in advance from the video image data, so that it is possible to provide a service that intends to provide a safe and secure society.
Functional Configuration
The communication unit 11 is a processing unit that controls communication with another device and is implemented by, for example, a communication interface or the like. For example, the communication unit 11 receives video image data or the like from each of the cameras 2, and outputs a processing result or the like obtained by the information processing apparatus 10 to a device or the like that is designated in advance.
The storage unit 20 is a processing unit that stores therein various kinds of data, a program executed by the control unit 30, or the like and is implemented by, for example, a memory, a hard disk, or the like. The storage unit 20 stores therein a video image data DB 21, a training data DB 22, a relationship model 23, a skeleton recognition model 24, a facial expression recognition model 25, a facial expression recognition rule 26, a higher-level behavior specific rule 27, and a probability model 28.
The video image data DB 21 is a database that stores therein video image data that is captured by each of the plurality of cameras 2 that are installed in the store 1. For example, the video image data DB 21 stores therein video image data for each of the cameras 2 or for each period of time for which images are captured.
The training data DB 22 is a database that stores therein graph data and various kinds of training data, such as the skeleton recognition model 24, the facial expression recognition model 25, and the probability model 28, that are used to generate various machine learning models. The training data stored here includes supervised learning data to which correct answer information is added and unsupervised learning data to which correct answer information is not added.
The relationship model 23 is one example of a machine learning model for identifying a relationship between each of the target objects included in the video image data. Specifically, the relationship model 23 is a model that is used for HOID (human object interaction detection), that is generated by performing machine learning, and that identifies a relationship between a person and another person or a relationship between a person and a thing (object).
For example, when a relationship between a person and another person is specified, a model that is used for the HOID and that specifies and outputs, in accordance with an input of a frame included in the video image data, a first class that indicates a first person and first region information that indicates a region in which the first person appears, a second class that indicates a second person and second region information that indicates a region in which the second person appears, and a relationship between the first class and the second class is used as the relationship model 23.
Furthermore, when a relationship between a person and an object is specified, a model that is used for the HOID and that specifies and outputs a first class that indicates a person and first region information that indicates a region in which the person appears, a second class that indicates an object and second region information that indicates a region in which the object appears, and a relationship between the first class and the second class is used as the relationship model 23.
Furthermore, the relationship indicated here is only an example and includes not only a simple relationship, such as “hold”, but also a complex relationship, such as “hold a commodity product A in one's right hand”, “stalking a person walking ahead”, or “look over one's shoulder”. Furthermore, as the relationship model 23, it may be possible to separately use the above described two different models that are used for the HOID, or it may be possible to use a single model that is used for the HOID and that is generated to identify both of a relationship between a person and another person and a relationship between a person and an object. In addition, the relationship model 23 is generated by the control unit 30 that will be described later, but it may be possible to use a model that is generated in advance.
The skeleton recognition model 24 is one example of a machine learning model for generating skeleton information that is one example of a feature value of a person. Specifically, the skeleton recognition model 24 outputs two-dimensional skeleton information in accordance with an input of image data. For example, the skeleton recognition model 24 is one example of a deep learning device that estimates a two-dimensional joint position (skeleton coordinates), such as a head, a wrist, a hip, or an ankle, with respect to two-dimensional image data of a person and that recognizes a motion corresponding to a basic motion or recognizes a rule that is defined by a user.
By using the skeleton recognition model 24, it is possible to recognize a basic motion of a person and acquire a position of an ankle, an orientation of a face, and an orientation of the body. Examples of the basic motion include, walk, run, and stop. An example of the rule defined by the user includes a transition of the skeleton information that corresponds to each of behaviors that are performed before a behavior of picking up a commodity product in hand. Furthermore, the skeleton recognition model 24 is generated by the control unit 30 that will be described later, but it may be possible to use data that is generated in advance.
The facial expression recognition model 25 is one example of a machine learning model for generating facial expression information related to a facial expression that is one example of a feature value of a person. Specifically, the facial expression recognition model 25 is a machine learning model that estimates an action unit (AU) that is a method for disassembling and quantifying a facial expression on the basis of parts of a face and muscles of facial expression. The facial expression recognition model 25 outputs, in accordance with an input of image data, a facial expression recognition result, such as “AU 1: 2, AU 2: 5, AU 4: 1, . . . ”, that represents an occurrence intensity (for example, 5-grade evaluation) of each of the AUs from an AU 1 to an AU 28 that are set in order to specify a facial expression. Furthermore, the facial expression recognition model 25 is generated by the control unit 30 that will be described later, but it may be possible to use data that is generated in advance.
The facial expression recognition rule 26 is a rule for recognizing a facial expression by using an output result obtained from the facial expression recognition model 25.
The higher-level behavior specific rule 27 is a rule for specifying a current behavior performed by a person.
In the example illustrated in
Furthermore, each of the elemental behaviors is associated with a basic motion and a facial expression. For example, regarding the elemental behavior B, information indicating that, for the basic motion, “as a time series pattern in a period of time from a time t1 to a time t3, the basic motion of the entire body is transitioned to basic motions 02, 03, and 03, the basic motion of a right arm is transitioned to basic motions 27, 25, and 25, and the basic motion of a face is transitioned to basic motions 48, 48, and 48”, and information indicating that, for the facial expression, “as a time series pattern in a period of time from the time t1 to time t3, a facial expression H continues” are defined.
Furthermore, the representation, such as the basic motion 02, is denoted by using an identifier for identifying each of the basic motions in terms of explanation, and corresponds to, for example, stop, raise an arm, squat down, or the like. Similarly, the representation, such as the facial expression H, is denoted by using an identifier for identifying each of the facial expressions in terms of explanation, and corresponds to, for example, a smiling face, an angry face, or the like. Furthermore, the higher-level behavior specific rule 27 is generated by the control unit 30 that will be described later, but it may be possible to use data that is generated in advance.
The probability model 28 is one example of a probability model for predicting a future behavior or a future state of a person from the basic motion and the facial expression information. For example, in the probability model 28, it is possible to use a hidden Markov model (HMM) in which, in a circumstance in which a state is not able to be observed, a variable value that is observable depending on the state is determined. Furthermore, in the present embodiment, the state of the hidden Markov model corresponds to a future behavior, and the variable value corresponds to observation information that includes a current behavior (or a combination of the skeleton information and the facial expression) and a relationship. In addition, a transition probability is defined between each of the states, and a probability distribution representing a relationship between the state and the observation information is defined between the state and the observation information.
For example, in the hidden Markov model illustrated in
In other words, by using the hidden Markov model as described above, the information processing apparatus 10 is able to estimate a behavior by using the observation information obtained during a period of time from a past to the present, and predict a future behavior (behavior that will be performed in the future) with respect to the probability of the estimated behavior only from the transition probabilities obtained from the hidden Markov model.
Furthermore, the observation information that is used in the hidden Markov model is able to be arbitrarily changed. For example, it is possible to use, as the observation information, a “current behavior” and a “relationship” and adopt, as each of the states, the hidden Markov model that uses a “behavior of a prediction target”. In other words, if the observation information on “the current behavior and the relationship” obtained from the video image data is input, the hidden Markov model estimates a near future or a current “behavior” by using the observation information. Then, the information processing apparatus 10 predicts and outputs, in accordance with the hidden Markov model, a “behavior” with the highest transition probability on the basis of the estimated “behavior” as a future behavior.
In this way, the information processing apparatus 10 is able to use the hidden Markov model, re-predict the current behavior that is specified from the skeleton information or the like, and performs future behavior prediction after increasing reliability of the current behavior, so that it is possible to expect an increase in accuracy.
As another example, it is possible to apply a hidden Markov model that uses, as the observation information, “the skeleton information and the facial expression obtained up to the current” and a “relationship” and that uses, as each of the states, a “behavior of a prediction target”. In other words, if observation information on “the current skeleton information, the facial expression, and the relationship” obtained from the video image data is input, the hidden Markov model estimates a current “behavior” from the obtained observation information. Then, the information processing apparatus 10 predicts and outputs, in accordance with the hidden Markov model, a “behavior” with the highest transition probability as a future behavior on the basis of the estimated current “behavior”.
In this way, the information processing apparatus 10 is able to perform the future behavior prediction after having directly predicted the current behavior from the skeleton information or the like by using the hidden Markov model, so that it is possible to expect an increase in a prediction speed.
A description will be given here by referring back to
Pre-Processing Unit 40
The pre-processing unit 40 is a processing unit that generates each of the models, the rules, and the like by using the training data that is stored in the storage unit 20 before an operation of the behavior prediction is performed. The pre-processing unit 40 includes a relationship model generation unit 41, a skeleton recognition model generation unit 42, a facial expression recognition model generation unit 43, a rule generation unit 44, and a probability model generation unit 45.
Generation of Relationship Model
The relationship model generation unit 41 is a processing unit that generates the relationship model 23 by using the training data that is stored in the training data DB 22. Here, as one example, a case will be described as an example in which, as the relationship model 23, a model that is used for the HOID is generated by using a neural network or the like. Furthermore, a case will be described as an example in which a model that is used for the HOID for specifying a relationship between a person and an object is generated, but it is possible to generate, in a similar manner, a model that is used for the HOID for specifying a relationship between a person and another person.
First, training data that is used for a model for the HOID in a process of machine learning will be described.
In the correct answer information, a class (the first class) of a person who is a detection target, a class (the second class) of an object that is a target for a purchase or an operation performed by a person, a relationship class that indicates an interaction between a person and an object, and a bounding box (Bbox indicating region information on an object) that indicates a region of each of the classes are set. In other words, as the correct answer information, information on the object grasped by a person is set. Furthermore, the interaction between the person and the object is one example of a relationship between a person and an object. In addition, if the interaction is used to specify a relationship between one of the persons and the other of the persons, a class that indicates the other person is used as the second class, the region information on the other person is used as the region information on the second class, and a relationship between the one person and the other person is used as a relationship class.
In the following, machine learning performed on a model used for the HOID by using the training data will be described.
Generation of Skeleton Recognition Model 24
The skeleton recognition model generation unit 42 is a processing unit that generates the skeleton recognition model 24 by using the training data. Specifically, the skeleton recognition model generation unit 42 generates the skeleton recognition model 24 by performing supervised learning that is performed by using the training data with correct answer information (label) attached.
In addition, it is possible to use, for the training data, each of the pieces of image data to which “walk”, “run”, “stop”, “stand up”, “stand in front of a shelf”, “pick up a commodity product”, “turn one's neck to right”, “turn one's neck to left”, “look upward”, “tilt one's head downward”, or the like is added as the “label”. Furthermore, generation of the skeleton recognition model 24 is only one example, and it is possible to use another method. In addition, it may also be possible to use, as the skeleton recognition model 24, behavior recognition that is disclosed in Japanese Laid-open Patent Publication No. 2020-71665 and Japanese Laid-open Patent Publication No. 2020-77343.
Generation of Facial Expression Recognition Model 25
The facial expression recognition model generation unit 43 is a processing unit that generates the facial expression recognition model 25 by using the training data. Specifically, the facial expression recognition model generation unit 43 generates the facial expression recognition model 25 by performing supervised learning using training data to which correct answer information (label) is added.
In the following, generation of the facial expression recognition model 25 will be described with reference to
As illustrated in
In a process for generating training data, the facial expression recognition model generation unit 43 acquires image data that is captured by the RGB camera 25a and a result of the motion capture that is obtained by the IR camera 25b. Then, the facial expression recognition model generation unit 43 generates an occurrence intensity 121 of an AU and image data 122 in which markers are deleted from the image data on the captured image by performing image processing. For example, the occurrence intensity 121 may be data that represents the occurrence intensity of each of the AUs in five-grade evaluation using A to E, and to which annotation, such as “AU 1: 2, AU 2: 5, AU 4: 1, . . . ”, is added.
In a process of machine learning, the facial expression recognition model generation unit 43 performs machine learning by using the image data 122 and the occurrence intensity 121 of each of the AUs that are output from the generation process performed on the training data, and generates the facial expression recognition model 25 that is used to estimate an occurrence intensity of each of the AUs from the image data. The facial expression recognition model generation unit 43 is able to use the occurrence intensity of each of the AUs as a label.
In the following, arrangement of the cameras will be described below with reference to
Furthermore, a plurality of markers are attached to a face of the subject whose image is captured so as to cover the AU 1 to the AU 28. The positions of the markers are changed in accordance with a change in a facial expression of the subject. For example, a marker 401 is arranged in the vicinity of an inner corner of an eyebrow (glabella). In addition, a marker 402 and a marker 403 are arranged in the vicinity of the smile line (nasolabial fold). The markers may also be arranged on the skin associated with one or more AUs and motions of muscles of facial expression. In addition, the markers may also be arranged so as to avoid the skin on which a texture is largely changed due to wrinkles or the like.
Furthermore, the subject wears an instrument 25c to which reference point markers are attached on the outside of the face contour. It is assumed that the positions of the reference point markers attached to the instrument 25c are not changed even if the facial expression of the subject is changed. As a result, the facial expression recognition model generation unit 43 is able to detect a change in the positions of the markers attached to the face on the basis of a change in the relative position from each of the reference point markers. Furthermore, by setting the number of reference point markers to three or more, the facial expression recognition model generation unit 43 is able to specify the positions of the markers in a three-dimensional space.
The instrument 25c is, for example, a headband. Furthermore, the instrument 25c may be a VR headset, a mask made of a hard material, or the like. In this case, the facial expression recognition model generation unit 43 is able to use a rigid surface of the instrument 25c as the reference point markers.
Furthermore, when images are captured by the IR cameras 25b and the RGB camera 25a, the subject continuously changes the facial expression. As a result, it is possible to acquire, as an image, a state of a change in the facial expression along the time series. Furthermore, the RGB camera 25a may also capture a moving image. The moving image can be regarded as a plurality of still images that are arranged in time series. In addition, the subject may also freely change the facial expression or may also change the facial expression in accordance with a scenario that is determined in advance.
Furthermore, it is possible to determine the occurrence intensity of each of the AUs on the basis of an amount of movement of the respective markers. Specifically, the facial expression recognition model generation unit 43 is able to determine the occurrence intensity on the basis of an amount of movement of a marker calculated on the basis of the distance between a position that is set in advance as a determination criterion and each of the positions of the markers.
In the following, the movement of the markers will be described with reference to
In this way, the facial expression recognition model generation unit 43 specifies the image data in which a certain facial expression of the subject is captured and the intensity of each of the markers at that time of the facial expression, and generates training data with an explanatory variable of “image data” and an objective variable of “an intensity of each of the markers”. Then, the facial expression recognition model generation unit 43 generates the facial expression recognition model 25 by performing supervised learning using the generated training data. For example, the facial expression recognition model 25 is a neural network. The facial expression recognition model generation unit 43 changes a parameter of the neural network by performing machine learning on the facial expression recognition model 25. The facial expression recognition model 25 inputs the explanatory variable to the neural network. Then, the facial expression recognition model 25 generates a machine learning model in which a parameter of the neural network is changed such that an error between an output result that is output from the neural network and the correct answer data that is the objective variable is reduced.
Furthermore, generation of the facial expression recognition model 25 is only one example and it may be possible to use another method. In addition, it may also be possible to use, as the facial expression recognition model 25, behavior recognition that is disclosed in Japanese Laid-open Patent Publication No. 2021-111114.
Generation of Higher-Level Behavior Specific Rule 27
A description will be given here by referring back to
After that, the rule generation unit 44 specifies a transition of the elemental behaviors (a transition of the basic motions and a transition of the facial expressions) that are detected during a period of time before the behavior XX is performed. For example, the rule generation unit 44 specifies, as the elemental behavior B, “a transition of the basic motions of the entire body, a transition of the basic motions of the right arm, and a transition of the basic motions of the face in the period of time from time t1 to t3” and “continuation of the facial expression H in the period of time from time t1 to t3”. Furthermore, the rule generation unit 44 specifies, as the elemental behavior A, “a transition of the basic motions of the right arm and a change from the facial expression H to the facial expression I in a period of time from time t4 to t7”.
In this way, the rule generation unit 44 sequentially specifies, as a transition of the elemental behaviors that are performed during a period of time before the behavior XX is performed, the elemental behavior B, the elemental behavior A, the elemental behavior P, and the elemental behavior J in this order. Then, the rule generation unit 44 generates the higher-level behavior specific rule 27 in which the “behavior XX” is associated with a “transition from the elemental behavior B, toward the elemental behavior A, the elemental behavior P, and the elemental behavior J”, and then, stores the higher-level behavior specific rule 27 in the storage unit 20.
Furthermore, generation of the higher-level behavior specific rule 27 is only one example, and it may be possible to use another method or it may be possible to manually generate the higher-level behavior specific rule 27 by an administrator or the like.
Generation of Probability Model 28
The probability model generation unit 45 is a processing unit that generates the probability model 28 by using the training data that is generated by collecting past events, past experiences, or the like.
For example, the probability model generation unit 45 trains the HMM by updating, by using the training data, the probability distribution obtained from the observation information that indicates a transition probability, which is related to a behavior and probabilistically exhibits a state transition indicating a behavior that is likely to be exhibited after a certain behavior, a feature value (a combination of skeleton information and a facial expression or a current behavior) of the person that is included in the video image data, and a relationship.
Furthermore, it is possible to use various known methods for the training method. In addition, the probability model 28 is able to use not only the HMM but also various models that are able to predict a potential state from the information that is able to be observed.
Operation Processing Unit 50
A description will be given here by referring back to
The acquisition unit 51 is a processing unit that acquires video image data from each of the cameras 2 and that stores the video image data in the video image data DB 21. For example, the acquisition unit 51 may acquire the video image data from each of the cameras 2 at any time or at periodic intervals.
Specifying Relationship
The relationship specifying unit 52 is a processing unit that performs a relationship specifying process for, by using the relationship model 23, specifying a relationship between a person and another person who appear in the video image data or a relationship between a person and an object that appear in the video image data. Specifically, the relationship specifying unit 52 inputs, for each frame included in the video image data, each of the frame to the relationship model 23, and specifies a relationship in accordance with an output result from the relationship model 23. Then, the relationship specifying unit 52 outputs the specified relationship to the behavior prediction unit 54.
As a result, for example, the relationship specifying unit 52 specifies, as the class of the person, a “person (customer)”, a “person (store clerk)”, or the like, and specifies a relationship indicating that “the store clerk is talking with the customer” between the “person (customer)” and the “person (store clerk)”. The relationship specifying unit 52 performs the above described relationship specifying process on each of the subsequent frames, such as a frame 2 and a frame 3, so that the relationship specifying unit 52 specifies, for each frame, a relationship of “talk”, a relationship of “hand over”, or the like.
In addition, as another example, the relationship specifying unit 52 inputs a frame to the relationship model 23 that has been subjected to machine learning, and specifies a class of a person, a class of an object, and a relationship between the person and the object. For example, the relationship specifying unit 52 specifies the “customer” as the class of the person, the “commodity product” as the class of the object, or the like, and specifies a relationship indicating that “the customer holds the commodity product” between the “customer” and the “commodity product”.
Specifying Current Behavior
The behavior specifying unit 53 is a processing unit that specifies a current behavior of a person from the video image data. Specifically, regarding each of the frames included in the video image data, the behavior specifying unit 53 acquires the skeleton information on each of the parts of a person by using the skeleton recognition model 24 and specifies a facial expression of the person by using the facial expression recognition model 25. Then, the behavior specifying unit 53 specifies a behavior of the person by using the skeleton information on each of the parts of the person that is specified with respect to each of the frames and the facial expression of the person, and outputs the specified skeleton information and the facial expression to the behavior prediction unit 54.
The behavior specifying unit 53 performs the above described specifying process on each of the subsequent frames, such as the frame 2 and the frame 3, and specifies, for each of the frames, the motion information on each of the parts of the person and the facial expression of the person who appears in the frame.
Then, the behavior specifying unit 53 performs the above described specifying process on each of the frames, so that the behavior specifying unit 53 specifies a transition of the motions of the respective parts of the person and a transition of the facial expressions. After that, the behavior specifying unit 53 compares the transition of the motions of the respective parts of the person and the transition of the facial expressions to each of the elemental behaviors in the higher-level behavior specific rule 27, and specifies the elemental behavior B.
Furthermore, the behavior specifying unit 53 specifies a transition of the elemental behaviors by repeatedly performing the process for specifying the elemental behavior from the video image data. Then, the behavior specifying unit 53 compares the transition of the elemental behaviors in the higher-level behavior specific rule 27, so that the behavior specifying unit 53 is able to specify the current behavior XX of the person appearing in the video image data.
Furthermore, in the example illustrated in
After that, similarly to
Prediction of Future Behavior
The behavior prediction unit 54 is a processing unit that performs future behavior prediction on a behavior of a person by using the current behavior of the person and the relationship. Specifically, the behavior prediction unit 54 inputs, to the probability model 28, the relationship that is specified by the relationship specifying unit 52 and the current behavior that is exhibited by the person and that is specified by the behavior specifying unit 53, and then, predicts a future behavior of the person. Then, the behavior prediction unit 54 transmits the prediction result to a terminal for an administrator or displays the prediction result on a display or the like.
Specifically, if the observation information indicating that “a person holds a screwdriver” is obtained at the current time, the behavior prediction unit 54 estimates a behavior of “the person picking up the screwdriver” as the behavior exhibited at the current time by inputting, to the HMM, the pieces of observation information that are obtained during a period of time between the past and the present. After that, the behavior prediction unit 54 specifies, in accordance with only each of the transition probabilities in the HMM, among the transition probabilities of the behaviors exhibited between the current behavior of “the person picking up the screwdriver” and another behavior, the behavior of “the person tightening up a screw” that is the highest transition probability. As a result, the behavior prediction unit 54 predicts the behavior of “the person tightening up a screw” as a future behavior after a certain period of time.
Furthermore, it may be possible to use, for the observation information illustrated in
Furthermore, in
At this time, if the current behavior is specified by a first frame that is one example of the image data at a certain time, and if the relationship is specified by a second frame, the behavior prediction unit 54 determines whether or not a second frame is detected in a certain range corresponding to a certain number of frames or a certain period of time that is set in advance from the point of time at which the first frame is detected. Then, if it is determined, by the behavior prediction unit 54, that the second frame is detected in the certain range that is set in advance, the behavior prediction unit 54 predicts a future behavior or a future state of the person on the basis of the behavior of the person included in the first frame and the relationship included in the second frame.
In other words, the behavior prediction unit 54 predicts a future behavior or a future state of the person by using the current behavior and the relationship that are detected at certain timings that are close with each other to some extent. Furthermore, the range that is set in advance may be arbitrarily set, and either of the current behavior and the relationship may be specified first.
Flow of Process
Then, the operation processing unit 50 inputs the frame to the skeleton recognition model 24, and acquires the skeleton information that is related to the person and that indicates a motion of, for example, each of the parts (Step S104). Furthermore, if a person does not appear in the frame at Step S103, the operation processing unit 50 omits the process at Step S104.
Furthermore, the operation processing unit 50 inputs the frame to the facial expression recognition model 25, and specifies a facial expression of the person from the output result and the facial expression recognition rule 26 (Step S105). In addition, if a person does not appear in the frame at Step S103, the operation processing unit 50 omits the process at Step S105.
After that, the operation processing unit 50 specifies a corresponding elemental behavior from the higher-level behavior specific rule 27 by using the skeleton information on the person and the facial expression of the person (Step S106). At this time, if the current behavior of the person is not specified (No at Step S107), the operation processing unit 50 repeats the process at Step S101 and the subsequent process to be performed on a next frame.
In contrast, if the current behavior of the person is specified (Yes at Step S107), the operation processing unit 50 inputs the current behavior and the specified relationship to the probability model 28, and predicts a future behavior of the person (Step S108). After that, the operation processing unit 50 outputs a result of the behavior prediction (Step S109).
In the following, specific example of solutions that contribute to achievement of a safe and secure society produced by using the behavior prediction performed by the information processing apparatus 10 described above will be described. Here, a solution that uses a relationship between a person and an object and a solution that uses a relationship between a person and another person will be described.
Solution that Uses Relationship Between Person and Object
As illustrated in
Furthermore, the information processing apparatus 10 performs skeleton recognition by using the skeleton recognition model 24, performs facial expression recognition by using the facial expression recognition model 25, and then, specifies, by using the recognition results thereof, the current behavior of the person A “holding the commodity product A”, the current behavior of the person B “pushing the cart”, the current behavior of the person C “walking”, and the current behavior of the person D “stopping”.
Then, the information processing apparatus 10 performs behavior prediction by using the current behaviors and the relationships, and predicts a future behavior of the person A indicating that the person A is “highly likely to purchase the commodity product A”, a future behavior of the person B indicating that the person B is “highly likely to perform shoplifting”, and a future behavior of the person C indicating that the person C is “highly likely to leave the store without purchasing anything”. Here, the relationship is not specified for the person D, so that the person D is excluded from the target of the behavior prediction.
In other words, the information processing apparatus 10 specifies a customer who moves in an area of a commodity product shelf that is a predetermined area of the video image data, specifies a target commodity product to be purchased by the customer, specifies, as the relationship, a type of a behavior (for example, watching, holding, etc.) of the customer exhibited with respect to the commodity product, and predicts a behavior (for example, purchasing, shoplifting, etc.) related to the purchase of the commodity product exhibited by the customer.
In this way, the information processing apparatus 10 is able to make good use of the above described behavior prediction for an analysis of a purchase behavior, such as a behavior or a route that leads to a purchase, a purchase marketing, or the like. Furthermore, the information processing apparatus 10 is able to detect a person, such as the person B, who is likely to commit a crime of, for example, shoplifting and is able to make good use of preventing a crime by strengthening surveillance of the person.
Solution that Uses Relationship Between Person and Another Person
As illustrated in
Furthermore, the information processing apparatus 10 performs skeleton recognition by using the skeleton recognition model 24, performs facial expression recognition by using the facial expression recognition model 25, and specifies, by using the recognition results thereof, the current behavior of the person A “walking ahead of the person B” and the current behavior of the person B “hiding away”.
Then, the information processing apparatus 10 predicts, on the basis of the behavior prediction performed by using the current behavior and the relationship, a future behavior of the person A indicating that the person A is “highly likely to be attacked by the person B”, and the future behavior of the person B indicating that the person B is “highly likely to attack the person A”.
In other words, the information processing apparatus 10 is able to predict a criminal act of the person B performed with respect to the person A by assuming that the person A is a victim, the person B is a committer, on the basis of the relationship of “stalking” of the committer with respect to the victim. As a result, the information processing apparatus 10 is able to detect a location where a crime is likely to be committed on the basis of a result of the above described behavior prediction, and implement a preventive measure, such as calling the police or the like. Furthermore, it is possible to make good use of examination of countermeasures, such as an increase in street lights.
Effects
As described above, the information processing apparatus 10 is able to predict a sign, instead of an occurrence of an accident or a crime, so that the information processing apparatus 10 is able to detect, from the video image data, a situation in which a countermeasure is needed in advance. Furthermore, the information processing apparatus 10 is able to perform behavior prediction from the video image data that is captured by a commonly used camera, such as a monitoring camera, so that the information processing apparatus 10 may be introduced into an existing system without a need of a complicated system configuration of a new device. In addition, the information processing apparatus 10 is introduced into an existing system, so that it is possible to reduce a cost as compared to a case in which a new system is constructed. Furthermore, the information processing apparatus 10 is able to predict not only simple behaviors that are continued from the past and current behaviors but also complicated behaviors of a person that are not able to simply specify from the past and current behaviors. As a result, the information processing apparatus 10 is able to improve prediction accuracy of a future behavior of a person.
Furthermore, the information processing apparatus 10 is able to implement the behavior prediction by using two-dimensional image data without using three-dimensional image data, so that it is possible to increase a speed of a process as compared to a process performed by using a laser sensor or the like that is recently used. In addition, the information processing apparatus 10 is able to rapidly detect, with a high-speed process, a situation in which a countermeasure is needed in advance.
In the above explanation, a description has been given of the embodiments according to the present invention; however, the present invention may also be implemented with various kinds of embodiments other than the embodiments described above.
Numerical Value, Etc.
Examples of the numerical values, the number of cameras, the label names, examples of the rules, examples of the behaviors, examples of the states, and the like used in the embodiment described above are only examples and may be arbitrarily changed. Furthermore, the flow of the processes described in each of the flowcharts may be changed as long as the processes do not conflict with each other. In addition, in the embodiment described above, the store is used as an example for the explanation; however, the example is not limited to this and may be applied to, for example, a warehouse, a factory, a classroom, the inside of a train, a passenger cabin of an airplane, or the like. In addition, the relationship model 23 is an example of a first machine learning model, the skeleton recognition model 24 is an example of a second machine learning model, and the facial expression recognition model 25 is an example of a third machine learning model.
System
The flow of the processes, the control procedures, the specific names, and the information containing various kinds of data or parameters indicated in the above specification and drawings can be arbitrarily changed unless otherwise stated.
Furthermore, the components of each unit illustrated in the drawings are only for conceptually illustrating the functions thereof and are not always physically configured as illustrated in the drawings. In other words, the specific shape of a separate or integrated device is not limited to the drawings. Specifically, all or part of the device can be configured by functionally or physically separating or integrating any of the units depending on various loads or use conditions.
Furthermore, all or any part of each of the processing functions performed by the each of the devices can be implemented by a CPU and by programs analyzed and executed by the CPU or implemented as hardware by wired logic.
Hardware
The communication device 10a is a network interface card or the like and communicates with another device. The HDD 10b stores therein the programs and DBs that operate the functions illustrated in
The processor 10d operates the process that executes each of the functions described above in
In this way, the information processing apparatus 10 is operated as an information processing apparatus that performs a behavior prediction method by reading and executing the programs. Furthermore, the information processing apparatus 10 is also able to implement the same functions as those described above in the embodiment by reading the above described programs from a recording medium by a medium reading device and executing the read programs. Furthermore, the programs described in another embodiment are not limited to be executed by the information processing apparatus 10. For example, the above described embodiments may also be similarly used in a case in which another computer or a server executes a program or in a case in which another computer and a server cooperatively execute the program with each other.
The programs may be distributed via a network, such as the Internet. Furthermore, the programs may be executed by storing the programs in a recording medium that can be read by a computer readable medium, such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), a digital versatile disk (DVD), or the like, and read the programs from the recording medium by the computer.
According to an aspect of one embodiment, it is possible to detect, from video image data, a situation in which a countermeasure is needed in advance.
All examples and conditional language recited herein are intended for pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-215309 | Dec 2021 | JP | national |
