Patent Application
20210286983
The embodiments discussed herein are related to an estimation method, an estimation program, and an estimation device.
Traditionally, there has been a device that recognizes a posture and a movement of a person on the basis of a distance image (hereinafter, also referred to as depth image) output from a distance sensor (hereinafter, also referred to as depth sensor) that measures a distance to a person. The device extracts a skeletal model having a three-dimensional skeleton position, for example, on the basis of a distance image output from one distance sensor. Thereafter, the device specifies a motion of the person on the basis of the extracted skeletal model.
U.S. Patent Application Publication No. 2010/0197390, Japanese Laid-open Patent Publication No. 2017-119102, and International Publication Pamphlet No. WO 2017/187641 are disclosed as related art.
M. Kitagawa and B. Windser, “MoCap for Artists”, Elsevier/Focal Press, 2008, pp. 188-193, Yuji Oshima et al., “Three-dimensional dynamics focusing on lower trunk motion in the horizontal plane during sprinting”, Japan Journal of Physical Education, Health and Sport Sciences, vol. 61, no. 1, p. 115-131, June 2016, Shuta HASEGAWA et al. “Proposal and evaluation of device operation using hand gestures considering posture”, IPSJ SIG Technical Report, vol. 2012-HCI-147, no. 24, p. 1-6, March 2012 are disclosed as related art.
According to an aspect of the embodiments, an estimation method in which a computer executes processing includes: acquiring a first distance image that includes information regarding a distance from a sensor to a first subject; estimating three-axis polar coordinates data of the first subject from an acquired first distance image using a prediction model for posture recognition that has learned three-axis polar coordinates data based on a spine vector that corresponds to a spine of a second subject and a shoulder vector that corresponds to a line that connects both shoulders of the second subject that are generated on the basis of coordinate data that represents a position of the second subject and a second distance image based on the coordinate data of the second subject and the distance from the sensor; and estimating a posture of the first subject on the basis of the three-axis polar coordinates data of the first subject.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
To assist judges and strengthen athletes in scoring assistance competitions such as the gymnastics, establishment of a technique for recognizing three-dimensional complex movements of a skeleton of an athlete is required. A posture of a person is detected from a distance image, and skeleton recognition is performed by using a prediction model for skeleton recognition according to the posture of the person and the distance image. In traditional detection of the posture of the person, the posture of the person has been detected using two axes including a rotation axis that represents a direction of a line connecting both shoulders in the rotation direction and an inverted axis representing a direction of the spine in the forward roll direction. However, in a case where these two axes are used, a cartwheel direction of the person is not determined. Furthermore, when the person falls sideways, if the left and right shoulders are projected on a horizontal plane, the left and right shoulders overlap. Therefore, a component of a vector is eliminated, and an angle in the rotation direction is not obtained. Therefore, it is difficult to estimate the posture reflecting somersaults and twists in which these postures exist.
One aspect provides an estimation method, an estimation program, and an estimation device that estimate a posture reflecting somersaults and twists.
Hereinafter, embodiments of an estimation method, an estimation program, and an estimation device disclosed in the present application will be described in detail with reference to the drawings. Note that the present embodiment does not limit the disclosed technology. Furthermore, the following embodiments may be appropriately combined in a range where no inconsistency occurs.
The estimation system 1 is an example of a system that measures a person 5 who is a subject with the distance sensor 10 and estimates a posture or the like of the person 5 by the estimation device 100 on the basis of a measurement result.
For example, the distance sensor 10 measures (senses) a distance to an object using an infrared laser or the like for each pixel and outputs a distance image. The distance image includes a distance to each pixel. That is, for example, the distance image is a depth image representing a depth of the subject viewed from the distance sensor (depth sensor) 10. The distance sensor 10 measures, for example, a distance to an object (subject) at a distance up to about 15 m. In the present embodiment, the object is the person 5.
Here, with reference to
Returning to the description of
Next, a functional configuration of the estimation device 100 will be described with reference to
The communication unit 110 is implemented by, for example, a network interface card (NIC) or the like. The communication unit 110 is a communication interface that is wiredly or wirelessly connected to the distance sensor 10 and controls information communication with the distance sensor 10.
The display unit 111 is a display device to display various types of information. The display unit 111 is implemented by, for example, a liquid crystal display or the like as a display device. The display unit 111 displays various screens such as a display screen input from the control unit 130.
The operation unit 112 is an input device that receives various operations from a user of the estimation device 100. The operation unit 112 is implemented by, for example, a keyboard, a mouse, or the like as an input device. The operation unit 112 outputs an operation input by the user to the control unit 130 as operation information. Note that the operation unit 112 may be implemented by a touch panel or the like as an input device, and the display device of the display unit 111 and the input device of the operation unit 112 may be integrated.
The storage unit 120 is implemented by, for example, a storage device such as a semiconductor memory element such as a random access memory (RAM), a flash memory, or the like, a hard disk, an optical disk, or the like. The storage unit 120 includes a prediction model for posture recognition storage unit 121 and a prediction model for skeleton recognition storage unit 122. Furthermore, the storage unit 120 stores information used for processing by the control unit 130.
The prediction model for posture recognition storage unit 121 stores prediction model information that is used when a posture of the person 5 who is the first subject in the first distance image is determined. The prediction model for posture recognition storage unit 121 stores, for example, a prediction model for posture recognition that is a learned model that has performed machine learning on a distance image of a person who is the second subject and the three-axis polar coordinates data in association with each other. Note that the prediction model for posture recognition may be a learned model that has performed machine learning on the distance image of the person who is the second subject and a posture number obtained by classifying the three-axis polar coordinates data in association with each other. As an algorithm of machine learning, for example, random forest and deep learning can be used.
The prediction model for skeleton recognition storage unit 122 stores prediction model information indicating a hypothetical joint position (skeleton position) for each recognition result of the posture recognition. As the recognition result of the posture recognition, for example, the three-axis polar coordinates data or the posture number obtained by classifying the three-axis polar coordinates data can be used. The prediction model for skeleton recognition storage unit 122 associates, for example, a posture number, a distance image corresponding to the posture number, and information regarding the joint position of the person (prediction model information for skeleton recognition). Although not illustrated, it is assumed that the prediction model information for skeleton recognition corresponding to each posture number exist respectively. Note that the prediction model for skeleton recognition storage unit 122 is generated by performing machine learning about the three-axis polar coordinates data, the distance image, and the information regarding the joint position. Furthermore, the prediction model for skeleton recognition storage unit 122 may be generated by performing machine learning about various distance images corresponding to the posture number and the information regarding the joint position of the person. In this case, as an algorithm of machine learning, for example, random forest and deep learning can be used.
The control unit 130 is implemented, for example, by executing a program stored in an internal storage device by a central processing unit (CPU), a graphics processing unit (GPU), a micro processing unit (MPU), or the like using a RAM as a working region. Furthermore, the control unit 130 may be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.
The control unit 130 includes a learning unit 131, an acquisition unit 132, an estimation unit 133, and a recognition unit 134 and realizes or executes functions and actions of information processing to be described below. In other words, for example, each processing unit of the control unit 130 executes learning processing and estimation processing. Note that an internal configuration of the control unit 130 is not limited to the configuration illustrated in
Regarding a person who is the second subject to be learned, the learning unit 131 learns three-axis polar coordinates data representing a posture of the person and a distance image and generates a prediction model for posture recognition. For example, the learning unit 131 may acquire coordinate data generated by motion capturing the person who is the second subject from an information processing device for motion capturing (not illustrated). For example, the learning unit 131 may acquire coordinate data of the person who is the second subject generated using computer graphics (CG) from an information processing device for CG data creation. The acquired coordinate data is coordinate data representing a position of the person who is the second subject on three axes including the x, y, and z axes.
The learning unit 131 calculates a spine vector from two joints on the spine of the person who is the second subject on the basis of the acquired coordinate data. The learning unit 131 calculates an angle θ formed by the y axis and the calculated spine vector. The learning unit 131 projects the spine vector on an xz plane and calculates an angle φ formed by the spine vector and the z axis.
The learning unit 131 calculates a shoulder vector from two joints of the right shoulder and the left shoulder of the person who is the second subject on the basis of the acquired coordinate data. The learning unit 131 obtains a rotation matrix such that the spine vector overlaps the y axis. The learning unit 131 rotates the shoulder vector using the obtained rotation matrix. The learning unit 131 projects the rotated shoulder vector on the xz plane and calculates an angle ξ formed by the shoulder vector and the x axis. In other words, for example, the learning unit 131 calculates three-axis polar coordinates data having the angles θ, φ, ξ and as components.
The learning unit 131 calculates a distance of the person who is the second subject for each pixel of the distance image in a case where the distance sensor 10 is assumed, on the basis of the acquired coordinate data. The learning unit 131 generates the second distance image on the basis of the acquired coordinate data and the calculated distance from the distance sensor 10 for each pixel. That is, for example, the second distance image is a distance image in a case where it is assumed that the person who is the second subject to be learned be measured by the distance sensor 10.
The learning unit 131 learns the generated second distance image and the calculated three-axis polar coordinates data and generates a prediction model for posture recognition. The learning unit 131 stores the generated prediction model for posture recognition in the prediction model for posture recognition storage unit 121.
In other words, for example, the learning unit 131 learns the three-axis polar coordinates data of the second subject or the posture number obtained by classifying the three-axis polar coordinates data and the second distance image and generates the prediction model for posture recognition. Here, the three-axis polar coordinates data of the second subject is three-axis polar coordinates data based on a spine vector corresponding to the spine of the second subject and a shoulder vector corresponding to a line that connects both shoulders of the second subject that have been generated on the basis of coordinate data representing a position of the second subject. Furthermore, the second distance image is a distance image based on the coordinate data representing the position of the second subject and the distance from the distance sensor 10.
Here, with reference to
That is, for example, the three-axis polar coordinates expression 42 copes with a cartwheel of the person by dividing a value of the spine vector 38 into an inclined direction and an inclined amount. Furthermore, the three-axis polar coordinates expression 42 copes with a posture at which a person falls sideways by setting an axis when the direction of the shoulder vector 41 is calculated as the spine. When this is applied to gymnastics, the angle θ indicates a rotation component at the time when a somersault is performed, and the angle ξ indicates a rotation component at the time when a twist is performed. In other words, for example, the spine vector 38 represents an inclined direction and an inclined amount of the second subject, and the shoulder vector 41 represents the rotation direction of the second subject around the spine vector 38 as an axis.
The spine vector 38a and the shoulder vector 41a in the Euler angle expression 48 are respectively equal to the spine vector 38 and the shoulder vector 41 in the three-axis polar coordinates expression 42. The correspondence of the respective angles in the three-axis polar coordinates expression 42 and the Euler angle expression 48 are φ=φe, θ=θe, and ξ=ξe−φe. In gymnastics, because φe changes from moment to moment as the person moves, it is not possible to estimate a twist rotation amount using only ξe. Furthermore, ξ=ξe is not satisfied because, in the three-axis polar coordinates expression 42, the direction of the distance sensor 10 is defined as ξ=0°, and a reference of 0° is different from than in the Euler angle expression 48. Therefore, in a case where a somersault and a twist motion in the gymnastics are recognized, it can be said that the three-axis polar coordinates expression 42 is better than the Euler angle expression 48.
Returning to the description of
When the first distance image is input from the acquisition unit 132, the estimation unit 133 refers to the prediction model for posture recognition storage unit 121 and estimates the three-axis polar coordinates data of the person 5 who is the first subject using the prediction model for posture recognition. That is, for example, the estimation unit 133 recognizes the posture of the person 5. The estimation unit 133 outputs the first distance image and the estimated three-axis polar coordinates data to the recognition unit 134.
In other words, for example, the estimation unit 133 estimates the three-axis polar coordinates data of the first subject from the acquired first distance image, using the prediction model for posture recognition. Furthermore, the estimation unit 133 estimates a posture of the first subject on the basis of the three-axis polar coordinates data of the first subject. Furthermore, the estimation unit 133 outputs data regarding the estimated posture to skeleton recognition processing for recognizing a skeleton of the first subject using a prediction model for skeleton recognition selected on the basis of the estimated posture.
When the first distance image and the estimated three-axis polar coordinates data are input from the estimation unit 133, the recognition unit 134 refers to the prediction model for skeleton recognition storage unit 122 and selects the prediction model for skeleton recognition on the basis of the estimated three-axis polar coordinates data. The recognition unit 134 determines a three-dimensional position of the skeleton of the person 5 who is the first subject on the basis of the first distance image using the prediction model for skeleton recognition selected by the prediction model for skeleton recognition storage unit 122 and generates skeleton information indicating the posture of the person 5. That is, for example, the recognition unit 134 recognizes the skeleton of the person 5 who is the first subject. The recognition unit 134 displays the recognized skeleton information, for example, by outputting the skeleton information to the display unit 111. For example, the recognition unit 134 outputs an angle of each joint to the display unit 111 on the basis of the recognized skeleton information and makes the display unit 111 display the output angle. Furthermore, the recognition unit 134 may transmit an image of the recognized skeleton information and the angle of each joint to a terminal for a judge and display a scoring assistance screen on the terminal.
Note that the recognition unit 134 may output the generated skeleton information, for example, together with a distance image or a captured image captured by a camera, to a processing unit or a processing device that executes skeleton correction processing. That is, for example, the recognition unit 134 can output the generated skeleton information, for example, as a skeletal model, so as to be used for CG animation. Moreover, the recognition unit 134 may output the generated skeleton information to an external storage device (not illustrated) or the like, for example, by processing the skeleton information into a specific format. Note that the recognition unit 134 may generate a three-dimensional model on the basis of the generated skeleton information, and output the generated model to the display unit 111 and display the generated model on the display unit 111.
Next, an operation of the estimation device 100 according to the first embodiment will be described. First, the learning processing will be described with reference to
The learning unit 131 acquires coordinate data generated by motion capturing or CG processing on the person who is the second subject from an information processing device for motion capturing or CG processing (not illustrated). The learning unit 131 calculates a spine vector from two joints on the spine of the person who is the second subject on the basis of the acquired coordinate data (step S1). The learning unit 131 calculates an angle θ formed by the y axis and the calculated spine vector (step S2). The learning unit 131 projects the spine vector on an xz plane and calculates an angle φ formed by the spine vector and the z axis (step S3).
The learning unit 131 calculates a shoulder vector from two joints of the right shoulder and the left shoulder of the person who is the second subject on the basis of the acquired coordinate data (step S4). The learning unit 131 obtains a rotation matrix such that the spine vector overlaps the y axis and rotates the shoulder vector with the obtained rotation matrix (step S5). The learning unit 131 projects the rotated shoulder vector on the xz plane and calculates an angle ξ formed by the shoulder vector and the x axis (step S6).
The learning unit 131 calculates a distance of the person who is the second subject for each pixel of the distance image in a case where the distance sensor 10 is assumed, on the basis of the acquired coordinate data. The learning unit 131 generates a second distance image on the basis of the acquired coordinate data and the calculated distance from the distance sensor 10 for each pixel (step S7).
The learning unit 131 learns the generated second distance image and the calculated three-axis polar coordinates data (θ, φ, ξ) and generates a prediction model for posture recognition (step S8). The learning unit 131 stores the generated prediction model for posture recognition in the prediction model for posture recognition storage unit 121. As a result, the estimation device 100 can generate a prediction model for posture recognition that has learned the three-axis polar coordinates data and the second distance image of the second subject.
Subsequently, estimation processing for performing posture recognition on the first distance image acquired from the distance sensor 10 will be described.
The acquisition unit 132 receives a first distance image from the distance sensor 10 and starts acquisition (step S11). The acquisition unit 132 outputs the acquired first distance image to the estimation unit 133.
When the first distance image is input from the acquisition unit 132, the estimation unit 133 refers to the prediction model for posture recognition storage unit 121 and estimates the three-axis polar coordinates data of the person 5 who is the first subject using the prediction model for posture recognition (step S12). The estimation unit 133 outputs the first distance image and the estimated three-axis polar coordinates data to the recognition unit 134. In other words, for example, the estimation unit 133 is an example of an output control unit.
When the first distance image and the estimated three-axis polar coordinates data are input from the estimation unit 133, the recognition unit 134 refers to the prediction model for skeleton recognition storage unit 122 and selects a prediction model for skeleton recognition on the basis of the estimated three-axis polar coordinates data (step S13).
The recognition unit 134 generates skeleton information indicating the posture of the person 5 who is the first subject on the basis of the first distance image using the prediction model for skeleton recognition selected by the prediction model for skeleton recognition storage unit 122 (step S14). The recognition unit 134 outputs the recognized skeleton information and an angle of each joint on the basis of the skeleton information, for example, to the display unit 111 and makes the display unit 111 display the skeleton information and the angle (step S15). As a result, the estimation device 100 can output an estimation result regarding the skeleton of the first subject and the angle of each joint. In other words, for example, the estimation device 100 provides a method for evaluating the skeleton and the angle of each joint.
In this way, the estimation device 100 acquires a first distance image including information regarding a distance from the distance sensor 10 to the person 5 who is the first subject. Furthermore, the estimation device 100 estimates three-axis polar coordinates data of the first subject from the acquired first distance image using the prediction model for posture recognition that has learned the three-axis polar coordinates data of the second subject that is the person at the time of learning the prediction model for posture recognition and the second distance image. Here, the three-axis polar coordinates data of the second subject is three-axis polar coordinates data based on a spine vector corresponding to the spine of the second subject and a shoulder vector corresponding to a line that connects both shoulders of the second subject that have been generated on the basis of coordinate data representing a position of the second subject. Furthermore, the second distance image is a distance image based on the coordinate data representing the position of the second subject and the distance from the distance sensor 10. Furthermore, the estimation device 100 estimates the posture of the person 5 on the basis of the three-axis polar coordinates data of the person 5. As a result, the estimation device 100 can output an estimation result regarding the skeleton with higher accuracy by using the prediction model for skeleton recognition created for each posture.
Furthermore, the estimation device 100 outputs data regarding the estimated posture to skeleton recognition processing for recognizing a skeleton of the first subject using a prediction model for skeleton recognition selected on the basis of the estimated posture. As a result, the estimation device 100 can recognize the skeleton using the data regarding the posture.
Furthermore, in the estimation device 100, the spine vector represents the inclined direction and the inclined amount of the second subject, and the shoulder vector represents the rotation direction of the second subject around the spine vector as an axis. As a result, the estimation device 100 can generate and use the prediction model for posture recognition that has learned data of three-axis polar coordinates expression.
In the first embodiment described above, the skeleton recognition is performed using the estimated three-axis polar coordinates data. However, a technique (technique) may be determined by directly obtaining the number of times of somersault and the number of times of twist from the three-axis polar coordinates data, and an embodiment of this case will be described as a second embodiment. Note that, by denoting the components same as those of the estimation device 100 according to the first embodiment with the same reference numerals, description of the overlapped configuration and operation will be omitted.
The technique storage unit 223 stores the number of times of somersault and the number of times of twist in association with a technique (technique) of the gymnastics. Furthermore, the technique storage unit 223 may store the number of times of somersault, the number of times of twist, the technique of the gymnastics in association with a score.
The analysis result storage unit 224 stores angles θ and ξ for each frame of a first distance image from start of a technique to end of the technique and a motion analysis result of a person 5 who is a first subject.
The “frame number” is information indicating a frame number of the first distance image from the start to the end of the technique. That is, for example, the first distance image is a moving image corresponding to a time from the start to the end of the technique. The item “θ” is information indicating a value of an angle θ for each frame. The item “ξ” is information indicating a value of an angle ξ for each frame. Furthermore, a row following a frame number “final” is set as “motion analysis result”, and in items “θ” and “ξ” of the row, an analysis result of the angle θ and an analysis result of the angle ξ are respectively stored. In the example in
Returning to the description of
As the number of times of somersault determination processing, the determination unit 236 first sets an initial value to a variable θdiff used to calculate the number of times (θdiff=0). When the first distance image and the estimated three-axis polar coordinates data (θ, φ, ξ), are input from the estimation unit 133, the determination unit 236 stores the angle θ in a θ field of a corresponding frame number of the analysis result storage unit 224 and calculates θdiff=θdiff+(θ−θpre). Note that this calculation is also expressed as θdiff+=θ−θpre. Furthermore, θ is substituted for θpre (θpre=θ). Note that an initial value of θpre is zero. That is, for example, the determination unit 236 integrates an increase of the angle θ for each one frame with θdiff.
The determination unit 236 determines whether or not θdiff exceeds 180. That is, for example, the determination unit 236 determines whether or not θdiff exceeds 180°. In a case of determining that θdiff exceeds 180, the determination unit 236 increases the number of times of somersault by 0.5 times. Furthermore, the determination unit 236 subtracts 180 from θdiff (θdiff=θdiff−180) and determines whether or not the technique ends. Note that the subtraction is also expressed as θdiff−=180. In a case of determining that θdiff does not exceed 180, the determination unit 236 determines whether or not the technique ends without increasing the number of times of somersault.
In a case of determining that the technique does not end, the determination unit 236 similarly integrates an increase of the angle θ with θdiff for the next frame in the first distance image and increases the number of times of somersault according to the determination result whether or not θdiff exceeds 180. In a case of determining that the technique ends, the determination unit 236 determines the number of times of somersault at that time as the number of times of somersault of the technique (technique) and stores the number in a θ field corresponding to the motion analysis result of the analysis result storage unit 224.
As the number of times of twist determination processing, first, the determination unit 236 seta an initial value to a variable ξdiff used to calculate the number of times (ξdiff=0). When the first distance image and the estimated three-axis polar coordinates data (θ, φ, ξ) are input from the estimation unit 133, the determination unit 236 stores the angle ξ in a ξ field of the corresponding frame number of the analysis result storage unit 224 and calculates ξdiff=ξdiff+(ξ−ξpre). Note that this calculation is also expressed as ξdiff+=ξ−ξpre. Furthermore, is substituted for ξpre (86 pre=ξ). Note that the initial value of ξpre is zero. That is, for example, the determination unit 236 integrates the increase in the angle ξ for each one frame ξdiff.
The determination unit 236 determines whether or not ξdiff exceeds 180. That is, for example, the determination unit 236 determines whether or not ξdiff exceeds 180°. In a case of determining that ξdiff exceeds 180, the determination unit 236 increases the number of times of twist by 0.5 times. Furthermore, the determination unit 236 subtracts 180 from ξdiff (ξdiff=ξdiff−180) and determines whether or not the technique ends. Note that the subtraction is also expressed as ξdiff−=180. In a case of determining that ξdiff does not exceed 180, the determination unit 236 determines whether or not the technique ends without increasing the number of times of twist.
In a case of determining that the technique does not end, the determination unit 236 similarly integrates an increase of the angle ξ with ξdiff for the next frame in the first distance image and increases the number of times of twist according to the determination result whether or not ξdiff exceeds 180. In a case of determining that the technique ends, the determination unit 236 determines the number of times of twist at that time as the number of times of twist of the technique (technique) and stores the number in a ξ field corresponding to the motion analysis result of the analysis result storage unit 224. In other words, for example, the determination unit 236 determines at least one of the number of times of somersault and the number of times of twist of the first subject on the basis of a time-series change in the estimated three-axis polar coordinates data.
The determination unit 236 refers to the technique storage unit 223 and determines a technique on the basis of the determined number of times of somersault and the determined number of times of twist. Furthermore, the determination unit 236 may acquire a score of the determined technique (technique). The determination unit 236 outputs and displays the determined technique (technique), the number of times of somersault, and the number of times of twist, for example, to and on the display unit 111. Furthermore, the determination unit 236 may output and display the determined technique (technique), the number of times of somersault, the number of times of twist, and the score to and on the display unit 111. Furthermore, the determination unit 236 may transmit the determined technique (technique), the number of times of somersault, the number of times of twist, the score, and an image of the first subject captured by a camera (not illustrated) to a terminal for a judge and display a scoring assistance screen on the terminal.
Next, an operation of the estimation device 200 according to the second embodiment will be described. Note that, because learning processing is similar to that in the first embodiment, description thereof is omitted.
The determination unit 236 of the estimation device 200 executes the following processing subsequent to the processing in step S11. The determination unit 236 executes the number of times of somersault determination processing (step S21) and the number of times of twist determination processing (step S22) in parallel according to the first distance image input from the estimation unit 133 and the estimated three-axis polar coordinates data.
Here, the number of times of somersault determination processing will be described with reference to
When the first distance image and the estimated three-axis polar coordinates data (θ, φ, ξ) are input from the estimation unit 133, the determination unit 236 stores an angle θ in a θ field of a corresponding frame number of the analysis result storage unit 224 and calculates θdiff+=θ−θpre. Furthermore, θ is substituted for θpre (step S213).
The determination unit 236 determines whether or not θdiff exceeds 180 (step S214). In a case of determining that θdiff exceeds 180 (step S214: Yes), the determination unit 236 increases the number of times of somersault by 0.5 times. Furthermore, the determination unit 236 subtracts 180 from θdiff (step S215) and proceeds to step S216. In a case of determining that θdiff does not exceed 180 (step S214: No), the determination unit 236 proceeds to step S216 without increasing the number of times of somersault.
The determination unit 236 determines whether or not the technique ends (step S216). In a case of determining that the technique does not end (step S216: No), the determination unit 236 returns to step S212 and executes similar processing on the next frame of the first distance image. In a case of determining that the technique ends (step S216: Yes), the determination unit 236 determines the number of times of somersault at that time as the number of times of somersault of the technique (technique) (step S217), and returns to the original processing. As a result, the determination unit 236 can determine the number of times of somersault.
Next, the number of times of twist determination processing will be described with reference to
When the first distance image and the estimated three-axis polar coordinates data (θ, φ, ξ) are input from the estimation unit 133, the determination unit 236 stores an angle ξ in a ξ field of the corresponding frame number of the analysis result storage unit 224 and calculates ξdiff+=ξ−ξpre. Furthermore, ξ is substituted for ξpre (step S223).
The determination unit 236 determines whether or not ξdiff exceeds 180 (step S224). In a case of determining that ξdiff exceeds 180 (step S224: Yes), the determination unit 236 increases the number of times of twist by 0.5 times. Furthermore, the determination unit 236 subtracts 180 from ξdiff (step S225) and proceeds to step S226. In a case of determining that ξdiff does not exceed 180 (step S224: No), the determination unit 236 proceeds to step S226 without increasing the number of times of twist.
The determination unit 236 determines whether or not the technique ends (step S226). In a case of determining that the technique does not end (step S226: No), the determination unit 236 returns to step S222 and executes similar processing on the next frame of the first distance image. In a case of determining that the technique ends (step S226: Yes), the determination unit 236 determines the number of times of twist at that time as the number of times of twists of the technique (technique) (step S227), and returns to the original processing. As a result, the determination unit 236 can determine the number of times of twist.
Returning to the description of
In this way, the estimation device 200 determines at least one of the number of times of somersault and the number of times of twist of the first subject on the basis of a time-series change in the estimated three-axis polar coordinates data. As a result, the estimation device 200 can recognize the technique on the basis of the number of times of somersault and the number of times of twist.
Note that, in each of the embodiments described above, the person 5 who performs the gymnastics competition is measured, and the posture, the skeleton, and the motion are recognized. However, the present embodiment is not limited to this. For example, it can be applied to other scoring competitions such as figure skating, verification of movements during rehabilitation, analysis of forms of baseball, golf, basketball free throws, or the like, guidance assistance to new workers in line works in factories, or the like.
Furthermore, in each of the embodiments described above, a case where the number of the first subjects is one has been described. However, the present embodiment is not limited to this. For example, in a case of a plurality of persons such as basketball, women's rhythmic gymnastics groups, or the like, it is sufficient to capture a distance image by setting an angle of view that includes all the persons.
Furthermore, each of the components of each the units illustrated in the drawings does not necessarily need to be physically configured as illustrated in the drawings. In other words, for example, specific aspects of separation and integration of the respective components are not limited to the illustrated forms, and all or some of the components may be functionally or physically separated or integrated in an arbitrary unit depending on various loads, usage states, or the like. For example, the recognition unit 134 and a determination unit 135 may be integrated. Furthermore, the order of each illustrated processing is not limited to the order described above, and the processing may be concurrently executed or may be executed as changing the order in a range in which the processing content does not contradict.
Moreover, all or some of the various processing functions to be executed by each device may be executed by a CPU (or arithmetic devices such as GPU, MPU, or micro controller unit (MCU)). Furthermore, all or some of the various processing functions may of course be executed by a program to be analyzed and executed by a CPU (or arithmetic devices such as GPU, MPU, or MCU) or hardware using wired logic.
The various types of processing described in each embodiment described above can be implemented by executing a program which has been prepared in advance by a computer. Therefore, in the following, an example of the computer which executes a program having a similar function to each embodiment described above will be described.
As illustrated in
The hard disk device 308 stores an estimation program that has a function similar to each of the processing units including the learning unit 131, the acquisition unit 132, the estimation unit 133, and the recognition unit 134 illustrated in
For example, the input device 302 receives inputs of various types of information such as operation information from a user of the computer 300. For example, the monitor 303 displays various screens such as a display screen to the user of the computer 300. The interface device 305 is connected to, for example, a printing device or the like. For example, the communication device 306 has a function similar to the communication unit 110 illustrated in
The CPU 201 reads each program stored in the hard disk device 308 and develops and executes the program on the RAM 307 to execute various types of processing. Furthermore, these programs can make the computer 300 function as the learning unit 131, the acquisition unit 132, the estimation unit 133, the recognition unit 134 illustrated in
Note that it is not necessary for the estimation program described above to be stored in the hard disk device 308. For example, the program stored in a storage medium which can be read by the computer 300 may be read and executed by the computer 300. The storage medium that is readable by the computer 300 corresponds to, for example, a portable recording medium such as a compact disk read only memory (CD-ROM), a digital versatile disc (DVD), or a universal serial bus (USB) memory, a semiconductor memory such as a flash memory, a hard disk drive, or the like. Alternatively, the estimation program may be prestored in a device connected to a public line, the Internet, a LAN, or the like, and the computer 300 may read the estimation program from the device so as to execute the estimation program.
All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
This application is a continuation application of International Application PCT/JP2018/045976 filed on Dec. 13, 2018 and designated the U.S., the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2018/045976 | Dec 2018 | US |
| Child | 17335143 | US |
