Patent Grant
12257478
The invention relates to the technical field of golf, and in particular, to a method for assisting golf training and a computer program product.
With the promotion and popularization of golf in the world, more and more people like golf. However, due to the limitation of golf venues, high technical skills, high standardization of actions and great difficulty in learning, full-time coaches are usually needed, so the cost of the whole sports practice is relatively high. At present, in order to improve the training level and reduce the training cost, various golf simulators have emerged. One of the current mainstreams is to correct the actions through specific auxiliary devices; and the other is to record the actions of trainees through image recognition and compare these actions with standard actions of famous persons to correct them.
However, shortcomings in the use of the auxiliary devices include: discrepancy with the sports scenarios of actual golf venues, for example, various detection sensors need to be worn, which will lead to uncomfortable limb movements. In addition, a standard reference model used in current image comparison methods is usually relatively constant, under which reference to the actions of famous stars or fixed standard actions are generally made. However, it is difficult for different trainees to implement the actions of famous golf stars because of their different physical qualities and golf playing skills, which greatly affects the learning enthusiasm for the trainees; moreover, the forced imitation of the actions of the golf stars are also easy to cause sports injuries.
An embodiment of the invention provides a method for assisting golf training and a computer program product. An optimal personal model of a user is set and used to assist user training, so that the user can steadily improve golf playing skills on the premise that his/her physical conditions are satisfied.
In order to achieve the above objective, in an aspect, a method for assisting golf training is provided and includes:
Preferably, the method further includes: providing evaluation, diagnosis and/or correction suggestions for the current swing according to the action difference and/or the swing result difference.
Preferably, the method further includes: displaying the current swing action, the action difference and/or the swing result difference from a specified view according to received instructions.
Preferably, in the method,
Preferably, in the method, the analyzing a swing action in the video by using a plurality of preset human body key points related to the swing action to obtain an image frame sequence corresponding to each swing stage in the current swing and further obtain a human body key point set sequence corresponding to each image frame sequence includes:
Preferably,
Preferably, in the method, the displaying the current swing action, the action difference and/or the swing result difference from a specified view includes:
Preferably, in the method, the displaying and/or playing the swing action from the specified view includes:
Preferably, the method further includes one or more of the following steps:
In another aspect a computer program product is further provided, and includes a computer program, where when the computer program is executed by a processor, steps of the method for assisting golf training as described in any one of the foregoing are implemented.
In still another aspect, a device for assisting golf training is further provided and includes a memory and a processor, where the memory stores at least one and the at least one program is executed by the processor to implement steps of the method for assisting golf training as described in any one of the foregoing.
The technical solution has the following technical effects:
Furthermore, the method according to the embodiment of the invention can be used to provide the user with a staged test evaluation function, and the action training is associated with the effect data after the golf is hit by docking the golf training simulator, which can help the user to monitor the learning progress in real time and plan a training path scientifically.
To further illustrate various embodiments, the invention is provided with accompanying drawings. These accompanying drawings are part of the disclosure of the invention, are mainly used to illustrate the embodiment and can explain the operation principle of the embodiments together with relevant descriptions in the specification. With reference to these contents, ordinary persons skilled in the art should be able to understand other possible embodiments and the advantages of the invention. Components in the drawings are not drawn to scale, and similar component symbols are usually used to represent similar components.
The invention will now be further described with reference to the accompanying drawings and specific embodiments.
Data acquisition:
Such data acquisition includes acquisition of swing action video data by the video acquisition device and acquisition of golf hitting data and golf flight data by the golf simulator.
In a specific embodiment, swing action videos are not limited to video data acquired by an on-site video acquisition device, but also include uploaded video files pre-acquired, videos shot by high-definition cameras at different angles or videos shot by intelligent terminals such as mobile phones.
In a case that a PC terminal is used, one or more cameras (preferably, no more than six cameras are provided), motion sensors, gyroscopes and depth cameras can be connected to acquire swing data; in a case that mobile devices are used, the cameras and the gyroscopes from multiple angles can be connected through WIFI or a local area network (LAN). Club data, golf hitting data and data flight data are acquired by the high-speed camera of the simulator.
Data Analysis:
According to a relationship between the human body key points in the human body key point set sequence corresponding to each swing stage, a plurality of action features corresponding to a current swing action is determined.
In a specific embodiment, after the swing video of the user is accurately captured by the high-definition camera and the golf hitting data and the golf flight data are acquired by the simulator, the whole swing process is modeled and analyzed by the following steps.
(1) Extraction of Human Body Key Points
In the embodiment, 29 key points are marked for the human body in each frame of video images, as shown in
In a specific embodiment, human body and human body key point detection models, such as YoloPose, HRNet and PPDetection, are used for training to obtain the human body key point detection model according to the embodiment of the invention Mpose. The model Mpose is used for detecting the human body and human body key points in each image frame of swing videos, such as video V. Specifically, each frame extracted from the video V by decoding or a k(th) frame Framek obtained by sampling is subjected to inference operation and normalization operation Norm by using the model Mpose, to obtain a human body frame Rectk and a human body key point set Pk={p0, . . . , p28} in the image as shown in
Rectk,Pk=Norm(Mpose(Framek)).
Rectk, Pk includes the coordinates, width and height of the top left corner of the image where the frame is located, namely, [x, y, w, h]. The key point set Pk includes 29 key points {p0, . . . , p28}, as indicated by a dot as 2, and each key point is the coordinate position [x, y] of the point in the image.
Due to the problems such as shielding and angle in a detection process, Mpose predicted key points may have abnormal values or outliers. In order to solve the problems, preferably, a key point filtering algorithm is used to smooth Mpose the predicted key points. In an embodiment, for the Mpose predicted Rectk and Pk, a low-pass filtering operation H(••) and 2t of context information of Pk are used to perform filtering operation, so as to obtain a smooth predicted value (namely, a predicated point set P′k) corresponding to the human body key point set Pk, namely:
P′k=H([Pk−t, . . . ,Pk+t])
(2) Swing Action Sequence Detection
After the key point sequence of the video V is obtained through a key point extraction step, in order to detect the specific position of one swing action in the video, a model for detecting one swing action is obtained by training a sequence detection network based on a PoolFormer network structure MpF.
In this example, the video V may include one or more complete swing actions.
For the key point set sequence {P′0, . . . , P′n} of the video V obtained through key point extraction and filtering steps, the key point set sequence includes key point sets corresponding to n frames of images; the swing action is detected by using MpF, to obtain a key point sequence segment {S1, . . . , Sm} including a plurality of swing actions, and information included in Sk is a segment key point set sequence {P′q−r′, . . . , P′q−1′, P′q} in the swing action, namely, the key point sets in the (r+1) frames of images from the (q−r)th frame to the q(th) frame, where q and r are natural numbers; here, k is a variable and taken from a natural number from 1 to m; and the number of frames included in different Si can or may be different. When the video V includes only one swing action, the key point sequence segment detected MpF has only S1.
(3) Action Decomposition
In order to facilitate more detailed action comparison and analysis, preferably, the action sequences included in one swing action obtained through swing action sequence detection are aligned and normalized. Specifically, a standard complete swing sequence Ss={Pi, . . . , Pi+z} is pre-established and used as a length or frame number template to identify the number of frames included in one standard complete swing action. In this example, illustratively, z=240, that is, the number of action frames based on 240 frames. In order to align each element Sk of {S1, . . . , Sm} to the action sequences with the number of z, each element here is aligned to 240 frames, and an action alignment network is pre-trained Malign; and each swing action is aligned to a same length by using Malign, namely:
S′k=Malign(Sk).
Therefore, S′k also includes 240 key point sets S′k={Pk0, . . . , Pk239}.
In this example, the swing action is decomposed into the following five swing stages: Takeaway→Backswing→Downswing→Follow-Through→Finish.
Therefore, the key point sets {Pk0, . . . , Pk239} included in each swing action are intercepted and classified according to five types of five stages [Takeaway, Backswing, Downswing, Follow-Through, Finish], and an action classification model MPFtiny is trained by combining with PoolFormer-tiny. The trained action classification model MPFtiny is used for further detection and classification of S′k, to obtain the key point sequences of various stage actions SkTW, SkBS, SkDS, SkFT, and SkFN. Elements of SkTW, SkBS, SkDS, SkFT, and SkFN are composed of segment key point sets included in S′k. Refer to
(4) Feature Space Transformation and Calculation;
Preferably, in order to improve the comparison effect of the swing action in space and time, in the embodiment, a NeRF model MVPN is trained according to the key point attributes and video acquisition methods mentioned above, and a plurality of frame sequences can be mapped to a standard space by using the NeRF model MVPN. The NeRF model MVPN is used to convert {S1, . . . , Sm} for {S′1, . . . , S′m} detected in the images into {D1, . . . , Dm} or {D′1, . . . , D′m} in the standard space.
(5) Action Feature Extraction
A relationship between the sequence key points included in each stage of SkTW, SkBS SkDS, SkFT, and SkFN is calculated to calculate a plurality of action features corresponding to the current swing action, where the plurality of action features include: player height-to-body width ratio information, a head position, positions of five sense organs, a left shoulder apex position, a right shoulder apex position, a left elbow position, a right elbow position, a left wrist position, a right wrist position, a left hip position, a right hip position, a left knee position, a right knee position, a left heel position, a right heel position, a left tiptoe position, a right tiptoe position, a human body skeleton line, an Euler angle of the torso, a stance inclination angle, a human body revolving speed, traces of selected human body key points, traces of selected human body parts, a wrist swing trace and/or a swing speed; and the action features are selected according to actual needs of users and training.
Illustratively, a trace of a single key point TracejA can be obtained by extracting a trace sequence of the single key point pj selected in a swing action A, where:
TracejA={pj0A, . . . ,pjnA};
In this example, it is assumed that the swing action A includes n frames of images.
According to the selection and showing needs of the users, a trace of a plurality of or all key points selected or a trace of parts selected can be obtained, where the trace of parts selected is identified by the trace of key points on this part.
In other embodiments, as shown in
LF1=Line(pm1,pm2).
In an embodiment, according to the plan made by the coach and the selected areas of concern or indicators of concern such as parts of concern or measured values of concern including speed and maximum rotation angle, for example, when the change in a steering angle of a head needs to be calculated, a Euler angle formed by the human body key points 5, 6, 27 as shown in
Therefore, the corresponding action feature set can be extracted for a moment k:
Fk={fi|fi∈RN},N>40;
Illustratively, when more than 40 actions features are selected, N is greater than 40; in other embodiments, N may be less than or equal to 40 according to the number of action features selected.
Preferably, the action features of the current swing obtained from the above analysis are stored in the record corresponding to the current swing as the data of the current swing process.
Further, according to current swing data including the above action feature data, the golf hitting data, golf flight data and club data obtained by the simulator, the video picture of the current swing, the current swing is modeled, and the current swing data is stored as the data in the record of the current swing model. Further, each single swing of the user can be modeled accordingly, and the corresponding data can be stored in a model library as a record corresponding to the model.
(6) Action Comparison
The plurality of action features of the current swing is aligned with a plurality of action features corresponding to an optimal personal swing model preset for the user in time and space; one or more of the plurality of action features of the current swing is compared with corresponding one or more of the plurality of action features corresponding to an optimal personal swing model to obtain an action difference between the current swing and the optimal personal swing model, where the optimal personal swing model is preselected from historical swing actions of the user or is a swing action set by the user under the guidance of a coach; and
Specifically, due to the difference in shooting angles, preferably, the obtained action features are to normalized to the same scale as the selected model by a normalization method. Then, the action features are aligned to a same phase, that is, the action features are aligned to a predetermined swing length (namely, a frame number template F) in time and the center coordinates of the human body frame.
FNk (value after alignment)=Align (Normalize(Fk),F template)
In a specific embodiment, the alignment method Align ( ) is the nearest neighboring method, so that the actions acquired at each moment can match with the corresponding moment of the model.
Finally, the difference between a time line and space domain is calculated to obtain the comparison result, where the comparison result includes the action difference and the swing results; the action difference includes: an average distance of the selected human body key points at a corresponding moment, trace similarity of the selected human body key points or human body parts, a vector cosine distance of selected vectors, an angle change curve of selected angles, a maximum angle difference between selected angles, a maximum speed difference between selected speeds and/or a maximum difference between stance inclination angles; and the swing result difference includes: an average golf speed difference, a maximum golf speed difference, an average distance of golf hitting points, a golf serving angle difference, a golf flight distance difference and/or a golf rotation direction difference.
According to the embodiments of the invention, the difference between two swing actions is compared from the change in the action features in time and space.
For example, the relative degrees of the key points or the trace of two swing actions A, B can be compared. Specifically,
The trace sequence distance of all key points included in the swing actions A and B can be further calculated; and the mean distance of points trace (MDPT) of all key points included in the swing actions A and B can be further calculated:
According to the selection of the user, the trace sequence distance of the selected key points between two swing actions and the mean distance of points trace of all key points can be selected as the comparison result.
(7) Multi-View Comparison Presentation
Preferably, in order to reflect the real difference, the embodiment provides the function of showing the swing actions of the user and difference from multiple views.
In a specific embodiment, for each frame in the image frame sequence corresponding to the swing stage in the current swing, a multi-view video is generated by using a trained multi-view diffusion model SMD, where the multi-view diffusion model SMD is pre-trained by using a multi-view action picture pre-acquired by the video acquisition device, parameters of the video acquisition device, an example segmentation model SegNet for segmenting human body regions and human body joints; and after instructions are received, the swing action is displayed and/or played from the specified view.
Preferably, in order to reflect the comparison effect, in the embodiment of the invention, a predetermined generation model is, according to the action features of the current swing in the image frame sequence corresponding to each swing stage in the current swing and body parameters of the user, used to generate a translucent digital person swinging in a transparent background. Then, the SMD is used to reconstruct the digital person from a specific view. Finally, in a playing process, the reconstructed translucent digital person is superimposed in the reconstructed capture swing picture through a specific angle. As shown in
Preferably, the method according to the embodiment of the invention further includes: providing evaluation, diagnosis and/or correction suggestions for the current swing according to the action difference and/or the swing result difference; and updating the optimal personal swing model according to training situations and/or assessment situations of the user.
Preferably, the method according to the embodiment of the invention further includes: according to the difference of swing results and reset grading rules, grading the current swing action, where such grading reflects the level of the current swing.
In the embodiment, a user swings under the guidance of a coach; and the swing action of the user that satisfies the predetermined swing quality is set as the optimal personal model of the user. Under the optimal personal model, the coach can set up multiple training plans for trainees. In a training process, the trainees can judge their own training situations by comparing their own swing actions with those in the current optimal personal model. After the training plans are completed, a test can be set; the swing action of the user in the test is compared with that in the current optimal personal model to evaluate whether the user passes the test. If the user passes the test, the current optimal personal model can be updated according to a next training plan. In the embodiment of the invention, the comparison is made based on the optimal personal model formulated by the user under the guidance of professional coaches; in the technical solution, the physical condition, technical level and personal characteristic of the user will be fully considered, thereby providing the user with the training solution that is most suitable for the actual demand and achieving safe and gradual improvement for the user.
As shown in
The swing modeling according to the embodiment of the invention includes:
A single swing process is modeled by using relevant data of the single swing process of the user, such as action feature data, golf hitting data and golf flight data corresponding to the single swing; and the relevant swing data corresponding to the single swing process is stored in the model library as the record of the corresponding model. Specifically, the swing data corresponding to the single swing process is stored in each record in the model library, and includes, but is not limited to: a complete swing action picture, a golf hitting position, a golf serving position, a golf serving angle, a golf serving speed, a golf flight distance, a rotation direction, a rotation speed, player height-to-body width ratio information, a head position, positions of five sense organs, a left shoulder apex position, a right shoulder apex position, a left elbow position, a right elbow position, a left wrist position, a right wrist position, a left hip position, a right hip position, a left knee position, a right knee position, a left heel position, a right heel position, a left tiptoe position, a right tiptoe position, a human body skeleton line, an Euler angle of the torso, a stance inclination angle, a human body revolving speed, a wrist swing trace, a club head trace and/or a swing speed, and the like. The above records include the action feature of the single swing and the corresponding golf hitting data and golf flight data.
For the swing video or swing process selected as the optimal personal model, the data analysis similar to that in Embodiment 1 is also performed, and the above data related to the swing process in the optimal personal model is obtained and stored in the model library as the record of the optimal personal model. Corresponding record data can be directly extracted from the model library to compare two swing actions. Further, the data in the record can be used to calculate other data to be compared for analysis or comparison.
The invention further provides a device for assisting golf training, and the device includes a processor, a memory, and a computer program that is stored in the memory and can run on the processor; and when the processor executes the computer program, steps in the method according to the embodiment of the invention are implemented.
The method according to the embodiment of the invention can be implemented through software; and the software can be executed on a mobile terminal or a PC terminal.
The invention further provides a computer readable storage media, where a computer program is stored in the computer readable storage medium; and when the computer program is executed by a processor, steps of the method according to the embodiment of the invention are implemented.
The invention further provides a computer program product, and the computer program product includes a computer program; and when computer program is executed by a processor, steps of the method as described above are implemented.
Although the invention has been particularly shown and described with reference to preferred embodiments, it will be understood by those skilled in the art that various variations in form and details can be made in the invention without departing from the spirit and scope of the invention as defined in the appended claims, and all the variations fall within the protection scope of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202410864572.8 | Jun 2024 | CN | national |
This application is a continuation of U.S. patent application Ser. No. 18/788,219, filed Jul. 30, 2024, which claims priority to Chinese Patent Application 202410864572.8, filed on Jun. 28, 2024. U.S. patent application Ser. No. 18/788,219 and Chinese Patent Application 202410864572.8 are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9211439 | Pedenko | Dec 2015 | B1 |
| 9357329 | Jeffery | May 2016 | B2 |
| 9604142 | Bentley | Mar 2017 | B2 |
| 11132533 | Dreessen | Sep 2021 | B2 |
| 11638853 | Yang | May 2023 | B2 |
| 11941915 | Tyomkin | Mar 2024 | B2 |
| 11951372 | Tuxen | Apr 2024 | B2 |
| 11986719 | Vale | May 2024 | B2 |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18788219 | Jul 2024 | US |
| Child | 18824213 | US |
