Patent Application
20180050254
As an increasingly popular sport, golf has attracted millions of people around the world. Athletes and amateurs are always looking for ways to improve their skills. Sensor based golf coaching systems are commercially available. One such system provides an IMU (inertial measurement unit) sensor, denoted as M-Tracer™, on the golf club. The sensor tracks the golf club and outputs a high frequency swing trajectory as well as many other metrics such as impact speed, shaft angle etc. Although the sensor based golf coaching systems provide useful information, it is still difficult for a normal user to understand the information and link that information to his or her performance. It is within this context that the embodiments arise.
In some embodiments, a method for spatial alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis is provided. The method includes capturing inertial measurement data of a golf club swing through an inertial measurement unit (IMU), and sending the inertial measurement data of the golf club swing from the inertial measurement unit to a computing device. The computing device is configured to align a first axis of the inertial measurement data of the golf club swing to a first axis of further inertial measurement data from a mobile device, estimate translation of the inertial measurement data of the golf club swing to two-dimensional video of the golf club swing captured by the mobile device, align second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video, and overlay a projected golf club trajectory based on the aligned captured inertial measurement data of the golf club swing onto the two-dimensional video.
In some embodiments, a method for spatial alignment of golf-club inertial measurement data and two-dimensional video for golf club swing analysis, performed by a computing device is provided. The method includes receiving captured inertial measurement data of a golf club swing from an inertial measurement unit (IMU) attached to a golf club, and receiving, from a device having a video camera, a two-dimensional video of the golf club swing. The method includes determining translation of the inertial measurement data of the golf club swing to the two-dimensional video of the golf club swing. The method includes aligning first and second axes of the inertial measurement data of the golf club swing to first and second axes of the two-dimensional video, and overlaying a projected golf club trajectory onto the two-dimensional video, based on alignment of the inertial measurement data of the golf club swing and the two-dimensional video.
In some embodiments, a tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor, cause the processor to perform a method. The method includes receiving, from an inertial measurement unit (IMU) attached to a golf club, inertial measurement data of a golf club swing and receiving, from a mobile device, a two-dimensional video of the golf club swing in association with inertial measurement data relative to the mobile device. The method includes aligning a first axis of the inertial measurement data of the golf club swing to a first axis of the inertial measurement data relative to the mobile device, and determining translation of the inertial measurement data of the golf club swing to the two-dimensional video of the golf club swing. The method includes aligning second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video, and overlaying a projected golf club swing trajectory onto the two-dimensional video, based on alignment of the inertial measurement data of the golf club swing and the two-dimensional video.
Other aspects and advantages of the embodiments will become apparent from the following detailed description taken in conjunction with the accompanying drawings which illustrate, by way of example, the principles of the described embodiments.
The described embodiments and the advantages thereof may best be understood by reference to the following description taken in conjunction with the accompanying drawings. These drawings in no way limit any changes in form and detail that may be made to the described embodiments by one skilled in the art without departing from the spirit and scope of the described embodiments.
A golf coaching system for golf swing analysis performs spatial and temporal alignment of an inertial measurement unit (IMU) captured golf swing and a two-dimensional video of the golf swing, using an apparatus and various methods described herein. The methods can be performed on one or more processors, such as a processor of an IMU, a processor of a computing device and/or a processor of a mobile device (which could also be a computing device). One device that is suitable for performing portions of various methods and serving as a portion of a suitable apparatus is the M-Tracer™ of the assignee, which is an IMU that can be mounted to a golf club. The M-Tracer™ is equipped with wireless communication, and can send IMU data to another wireless device. Although embodiments are described herein using the M-Tracer™ as an IMU in one embodiment, it should be appreciated that variations and further embodiments are readily devised using other IMU systems, as the embodiments are not limited to the M-Tracer™ product.
With reference to
Also disclosed herein is a marker based method to spatially align the IMU system captured golf swing trajectory with a 2D video captured by a static regular camera that does not necessarily come with an embedded IMU sensor. The method has the following steps:
In US patent publication 2015/0072797, a system to overlay an IMU sensor captured golf swing trace onto a 2D video is provided. The method spatially aligns these two systems by requiring the golfer to stand at a predefined position. The system renders a human model on a preview screen and the golfer adjusts his or her position such that his or her posture aligns with the model. The human model is calculated based on information regarding the camera direction, such as front, back, etc., under the assumption that the golf player and camera are either parallel or perpendicular. The distance between the camera and golf player is calculated based on the information of the golf player's height and some pre-set parameters estimated from the Japanese population. It should be appreciated that assumption of camera position will not always be guaranteed and the pre-learned parameter may not be accurate for each case. If there exists a tilt in the camera or the distance between camera and player is inaccurate, the overlay of sensor captured swing trajectory will deviate from the actual swing motion from the video. In such a case, a manual adjustment has to be implemented through a user interface.
Present embodiments, however, provide a fully automatic solution to align the IMU system signal with video without requiring any additional human interaction, except for the optional user assisted mode in some embodiments. The golfer can stand at any position without pre-adjusting positions and the system does not require the camera to be positioned in parallel or perpendicular to the golf player. The method calibrates IMU system coordinate space and camera coordinate space by using IMU information from the smart phone and detecting image features, such as using a detected golf club line to align x-y plane rotation and using a determined golf ball size to determine distance between camera and golf player.
The embodiments align the IMU system trajectory with a 2D golf swing video by estimating the transformation from the IMU system coordinate system to the camera coordinate system, including three rotation angles (pitch θ, roll φ and yaw ψ) and three translation values in x, y and z axes.
The system takes the following as input: (1) IMU Reading: gravity sensor reading when the mobile device is static, (2) IMU system trajectory, (3) RGB (red, green, blue, i.e. color, or in further embodiments black and white) video from a mobile device, (4) camera intrinsics, and (5) temporal alignment information represented as time bias between the first frame of IMU system reading and the first RGB video frame. The temporal alignment is assumed to be solved by some method. A method of doing temporal alignment is provided in the present disclosure with reference to
Notations and Assumptions used herein are described below.
Let [xT, yT, zT]T denote a point represented in M-Tracer™, or other suitable IMU system, coordinate space and [xc, yc, zc]T denote the same point represented in camera space, thus
[xc,yc,zc]T=RMC[xT,yT,zT]T+TMC, (Eq. 1)
where RMC and TMC denote the rotation and translation from IMU system coordinate system to camera coordinate system.
Any 3D rotation can be represented by three Euler angles: pitch (θ), roll (φ) and yaw (ψ) which are defined as the rotation angle around x, y and z axis accordingly. Therefore, RMC can be further decomposed as
R
MC
=R(θ,φ)R(ψ) (Eq. 2)
In the following sections, the method to estimateR(θ, φ), R(ψ) and TMC is detailed. After obtaining RMC and TMC, IMU system trajectory can be transformed to camera coordinate system and its projection onto the 2D image plane [u, v] can be calculated accordingly using camera intrinsic parameters, i.e.,
where fx, fy denote the focal length measured in width and height of pixels and [cx, cy] denote the principal point coordinate.
There are some assumptions made by the proposed algorithm as follows:
(1) Camera is equipped with an IMU sensor that can measure the gravity direction relative to camera IMU coordinates.
(2) Camera is mounted to a fixed position and kept static during the golf club swing.
(3) During address time (i.e., the moment of golf club impact to the golf ball), golf club position is close to the golf ball.
Details of the Coordinate Systems are provided below. In order for the system to determine the transformation from the IMU system coordinate system to the camera coordinate system, three body coordinates are involved: IMU system CS (coordinate system), cell phone IMU CS and cell phone camera CS. These body coordinates are defined further below, in some embodiments. Note that in various embodiments, the coordinate system is defined as follows, but the algorithm does not require these specific coordinate systems, and conversions to other coordinate systems are readily devised so that different coordinate systems may be integrated with the embodiments.
Let [gxi, gyc, gzc]T denote the unit gravity_up vector represented in IMU coordinate system that can be obtained by normalizing the reading from gravity sensor, where gravity_up denotes the gravity direction pointing upward. Let [gxc, gyi, gzc]T denote the corresponding representation in camera coordinate system. According to
[gxc,gyc,gzc]T (Eq. 5)
where Rj2c denotes the rotation from IMU coordinate space to camera coordinate space, which can be obtained through a IMU-Camera calibration process such as the method of Lobo and Dias published in 2007. In most smart phones and tablets, the IMU chip is installed such that the three axes are aligned with camera axes as illustrated in
Let [gxT, gyT, gzT]T denote the unit gravity_up vector represented in the M-Tracer™ coordinate system. Aligning [gxc, gyc, gzc]T with [gxT, gyT, gzT]T is equivalent to find theR(θ,φ), such that
[gxc,gyc,gzc]T=R(θ,φ)[gxT,gyT,gzT]T (Eq. 6)
As depicted in
where R(θ, φ)13=−sin θ, R(θ, φ)23=sin φ cos θ, R(θ, φ)33=cos φ cos θ. Thus roll and pitch angle can be solved as
φ=a tan 2(gyc,gzc) (Eq. 8)
θ=a tan(−gxc,(gyc*sin φ+gzc*cos φ)) (Eq. 9)
Thus by setting ψ as an arbitrary value to be optimized later, e.g., ψ=0,R(θ,φ) can be calculated accordingly.
This method can be performed by Module 2: Calculating Translation. As shown in
T
MC
=[X
ball
,Y
ball
,Z
ball]T (Eq. 10)
Let uball, Vball and radball be the location and radius of the detected golf ball in a 2D image. Embodiments for the detection of ball location and ball size can be found later in the disclosure, with reference to
where fx, and fy denote focal length, cx, and cy the principle point of the camera that obtained from camera intrinsics and Rball is the priori known actual golf ball size in 3D.
It should be noted, if in another system, the origin of the M-Tracer™ (or other IMU system) coordinate system is not the golf ball position, it is easy to convert to the presently defined coordinate system by subtracting the whole IMU system trajectory by the value of the known position of club head position at impact time. It should also be noted that other objects around the ball (which have the same distance from camera) with known size could be used to estimate Zball. For instance:
To estimate yaw angle, two solutions are proposed. One solution is to maximize the projection similarity of the IMU system golf club line and the detected club line in frames of the video. The other solution is to minimize the error between the re-projected IMU system location and the detected IMU system location in 2D video. To further refine a result, an iterative process is applied to remove the outliers. In the following embodiment, the alignment is performed using the golf club line.
In an action 702, a golf club line is detected in multiple frames. In an action 704, the yaw angle is estimated according to equation 12. In an action 706, outliers are detected, and an outlier set is formed. In a determination action 708, it is determined whether an outlier set is empty. If the answer is yes, flow proceeds to the action 712, and the yaw angle is output. If the answer is no, flow proceeds to the action 710, to remove the outlier, and from there to the action 704 in order to iterate the estimation of the yaw angle.
If the outlier set is not empty in the determination action 708, flow alternatively could proceed to the action 714 to remove the outlier, and from there to action 718. Action 718 is also preceded by action 716, in which the golf club line is detected in multiple frames. In the action 718, the yaw angle is estimated according to equation 16 as an alternative to the action 704 using equation 12. Action proceeds from the action 718 to the action 706, to detect outliers.
The yaw angle is estimated by maximizing the projection similarity of the M-Tracer™ golf club line and the detected club in the frames. Assume that the detected line at frame i is li=[lx1i, ly1i, lx2i, ly2i] and the corresponding M-Tracer™ golf club line (i.e., the line connecting M-Tracer™ position to the club head position) is denoted by
mtli=[lx1i, ly1i, lz1i, lx2i, ly2i, lz2i].
The system solves the following maximization equation to find the yaw angle:
max argψΣi∉outliers sim(π(K,T,mtli),li), (Eq. 12)
where π is the image projection function, K is the camera intrinsic matrix, T is the final transformation matrix from the IMU system coordinate system to the camera system that is defined as follows:
and sim the similarity of the projected line and detected line at frame i. Explanation of each part is given below.
The image projection function, π(K, T, l), projects a 3D line segment 1=[lx1, ly1, lx2, ly2, lz2] determined by two points [lx1, ly1, lz1]T, [lx2, ly2, lz2]T into a 2D line segment lproj=[lx1proj, ly1proj, lx2proj, ly2proj]:
and sim as:
where . is the dot product.
An alternate method is to perform alignment with IMU system location. The yaw angle can also be estimated by minimizing the re-projection error of the IMU system location. To detect the IMU system device in the 2D video frames, a color or texture marker can be attached to the device. An alternative technique is to detect the LED light source on the device.
Let [XMTi, YMTi, ZMTi]T be 3D location of IMU system at time i represented in IMU system coordinate space and its projection to 2D image space, denoted as [] can be represented as
[xMTi,yMTi,zMTi]T=K(R(θ,φ)R(ψ)[XMTi,YMTi,ZMTi]T+TMC) (Eq. 15)
Let [uMTi, vMTi] be the detected IMU system location in 2D video frame, the yaw angle can be estimated by minimizing the reprojection error
min argψΣi∉Outliers∥(uMTi−,(vMTi−
)∥ (Eq. 16)
Where ∥.∥ denotes the norm 2 metric.
After initial estimation the yaw angle by either method described above, the results are further refined via an iterative outlier removal process, in some embodiments.
(1) Use N frames to estimate yaw angle and calculate transformation T=[R(θ, φ)R(ψ)|TMC]
(2) Find outliers, O, using estimated T
(3) Set new outliers, Onew=O
(4) Do the following until Onew is empty
To find outliers for a given transformation T (assuming that T is approximately correct), the system checks if the 2D detected lines are parallel to the 3D projected lines. This check is done by comparing the distance of each two ends of the projected 3D line segment to the corresponding 2D detected line. If the difference of the distances is higher than a threshold then the detected 2D line is an outlier.
Assume that lproj=[lx1proj, ly1proj, lx2proj, ly2proj] is the projection of a 3D line, l, using the operator π(K, T, l) and the corresponding 2D detected line is l2D=[lx12D, ly12D, lx22D, ly22D]. The outlier function ⊚ (l2D, lproj, ∈) returns 1 if the 2D detected line is outlier and is defined as:
where θ is the distance of a point to a line:
In an action 802, the video with the IMU system trajectory overlaid is showed. In a decision action 804, it is determined whether the user is satisfied. If yes, the flow ends. If no, flow proceeds to the action 806, to ask the user to click on a location of a ball (e.g., with a touchscreen or mouse or other user input). In an action 808, module 2 for ball detection is run on a window around the click location. In an action 810, module 3 for x y axis alignment is run. In an action 812, the video with the IMU system trajectory overlaid is illustrated. In a decision action 814, it is determined whether the user is satisfied. If yes, the flow ends. If no, flow proceeds to the action 816, to ask the user to click on the location of the IMU system in a new frame. In an action 818, a region of interest is defined around the click location for this frame, and made to the initial input four module 3. Flow proceeds to the action 810, to run module 3 again.
The above sections describe the method to align 2D video and IMU system signal by using a mobile device with an embedded IMU sensor. In this section, an alignment method with a regular camera that does not have an IMU sensor is described. The method performs the alignment by using a marker placed on the floor as shown in
A review of Eq. 1 shows the equation can be further extended as
RRC and TRC are rotation and translation form marker space to camera space that can be easily obtained by any marker detection algorithm. Thus the only remaining unknowns are RMR and TMR which are denoted as the rotation and translation from IMU system space to marker space.
TMR is the translation from the IMU system coordinate system to the marker coordinate system, which is the 3D location of the ball represented in marker coordinate space. This 3D location can be measured manually from the ball to the marker origin. An example is illustrated in
Another way to determine translation is to use the method described above with reference to
where TMC can be obtained following the method described with reference to
Due to the fact that the marker is placed on the floor, it is reasonable to assume that the negative Z-axis of marker space is the gravity_up direction that aligns with z-axis of the IMU system. The marker can be placed at a variety of locations as along as the marker plane is perpendicular to gravity. Thus,
where R(ψ) is the rotation around z-axis and ψ is the yaw angle. The estimation of R(ψ) can be done using method described with reference to
In order to detect a golf club in a frame, in one embodiment, first a background model is constructed. The background model is the average of the frames:
where s and e represent the first and last frames to average. The best range [s e] is the range covering fast moving frames (e.g., from golf club top position to impact) which can be obtained from the synchronized IMU system data. For the frame I first the background is subtracted and then the image is normalized:
Now the Canny edge detector and then the probabilistic Hough transform are applied to find lines. The lines are then filtered by following constraints:
Angle constraint: the angle of the line with x axis should be within the following range: [a−t a+t] or [−a−t−a+t], where a is the expected angle of the golf club with the x axis (e.g., 60°) and t the tolerance (e.g., 10°). Note, this constraint limits the detection of the golf club at some specific positions. However it helps to reduce many false positives. As only one parameter needs to be estimated, a couple of accurately detected lines will be sufficient to solve the unknown.
Color/Width constraint: the line should have the given minimum color with the minimum width. If multiple lines are detected, the longest line, ll=[lx1l, ly1l, lx2l, ly2l] (with [lx1l, ly1l] denoting the start point and [lx2, ly2l] the end point of the line), is considered as the golf club. The other lines, li, are compared with ll and if they are consistent (i.e., approximately parallel to ll and are within a specific distance), they are kept. Finally, ll and the consistent lines li are averaged as follows:
where [lx1avg, ly1avg] denotes the start point of the average line, [lx2avg, ly2avg], the end point of the average line, t the parametric value of the line and vi=[vxi, . . . vyi] the vector representing the direction:
This average line is computed for each given frame.
For golf club tracking, and in order to improve the robustness of the algorithm, in some embodiments the start and end points of the line along with their average (i.e., the middle point) are tracked by the pyramidal implementation using the Lucas-Kanade tracking method in one embodiment. If no line is detected but the points are tracked properly, the line is estimated by fitting a line to the points (e.g., by least square method).
For ball detection in some embodiments, in the 2D image the system first detects the golf club (see above) in the address pose (i.e., relatively static frames before the swing starts). Assume that the golf club is detected as [GCx1, GCy1, GCx2, GCy2], where [GCx1, GCy1] denote the upper side of the club (e.g., IMU system position) and [GCx2, GCy2] is the head of the club. The circular pattern centered at [x, y] and of radius of R is defined as:
The system searches for the circle pattern in a window W by extending the golf club from its head (Ext):
W={[x,y]∥[x,y]−Ext∥≦nr},(n≧1)
Ext={[x,y]=[(GCx2−v(1)·i)=R/2,(GCy2−v(2)·i)−R/2]},(a≦i≦b),
where, ∥ ∥ is L2 norm, a and b denote the search range along the extended line (i.e., represented by a parametric equation), and v the club line direction vector:
v=[GC
x1
−GC
x2
,GC
y1
−GC
y2]
To get the initial estimation, the system finds the circular pattern in window, W, by maximizing the cross correlation of the pattern and image I:
Once the initial center of the circle, [x, y], is found, a sub image at [x, y] with size of R is considered, the Canny edge detector and then the circle Hough transform are applied to find the center and radius of the golf ball (i.e., [xball, yball] and Radball). The golf ball may have a non-circular shape due to illumination conditions. Here, several methods are presented make ball detection and size estimation more robust.
Temporal alignment is described next. Since the IMU system and the camera system run on different clocks, a method is needed to align the timing of the systems together, i.e., temporal alignment. In this disclosure a method to automatically synchronize the timing of IMU system and the camera system is described. After synchronization, the IMU system captured signal (i.e., the IMU captured golf club swing trajectory) and video (i.e., the two-dimensional video of the golf club swing) are temporally aligned and can be further aligned spatially.
With reference to
A method of detecting impact time (i.e., the moment at which the golf club head impacts the golf ball) is as follows. The method compares the image intensity change of neighboring two frames (i−1th and ith frame) of a video of a golf club swing, at a determined ball location relative to each of the frames. If such change is larger than a threshold, the impact time is determined as the i−1th frame. This is further described below with reference to
Using intensity change only, to detect impact time usually suffers from noise from environmental factors such as lighting change and background object occlusion, resulting in false detection. Different from and improving upon the method above, a further method disclosed herein uses a temporal pattern to further verify the initial detection result from image intensity so that detection accuracy is improved. In addition, the method to synchronize the IMU system signal and video by using the detected impact frame is disclosed herein.
The system takes RGB video, IMU system trajectory with time stamp information and camera intrinsics as input. The system first detects a golf ball location from 2D video frame(s). Thus proceeding backwards from the last frame, where the ball is assumed to be outside of the view, the system checks the intensity value of the detected ball center, in each frame. If such intensity value is bigger than a threshold and is further verified by a temporal pattern, the current frame will be determined as the impact time frame. The initial time bias can then be calculated with a known IMU system impact time stamp as well as the video frame rate. Due to the nature of the method, the accuracy of the time bias is at most within 1 frame. A refinement process is followed to further improve the accuracy by shifting IMU system trajectory samples by Δt around the calculated time bias and finding the optimal Δt with the smallest spatial alignment error.
Image processing methods described above for detecting a ball-shaped object can be applied to detecting the golf ball from the 2D images in the video. In some embodiments, instead of detecting the ball appearance directly on a video frame, such detection is performed by the system on background subtracted images to reduce background noise. Let Vi denote the ith frame and li denote the background subtracted frames defined as
where ∥ is the absolute value and BG is the average of the last M (e.g., M=10) frames that we are sure the ball does not appear:
where N is the total number of frames.
Starting from the last frame of the video of the golf club swing, the system detects detect when there is a sudden intensity change in the location of the ball, i.e., check each frame at the ball location [u, v] to determine if the corresponding value is bigger than a threshold. The sudden intensity change in the ball location does not always suggest the impact time. In some cases, there is a possibility of noise or other objects appearing at the location of the ball and if the change in intensity is greater than the threshold, it may cause a false detection. An example of such a scenario is presented in
Therefore, in some embodiments, a verification step is followed after such an intensity change frame is found. The following pseudo code illustrates the process.
A threshold T can be defined as the average of maximum and minimum of the intensity value at ball location T=½ (max(li(u, v)+min(Ii(u, v)), for i=1:N
P
i
=[b
n
,n=i−A:i+B]
Thus the impact frame candidate will be determined as a true impact frame if (Pi, Ptmp)<TP, where D(.) is a distance metric and TP is a pre-defined threshold. In one example D (.) can be defined as Hamming distance.
The system estimates initial time bias as follows. Let fv and fM denote the frame rate of video and IMU system signal, let iv and im denote the impact frame detected from video and impact sample detected from the IMU system. Thus the time bias tdiff=ivfv−imfm.
However, in various embodiments of the system, the time bias should be refined. Due to the finite frame rate of the video, the highest accuracy of temporal synchronization based on the above approach is within ±1 video frames. In some embodiments, the frame rate (i.e., the sampling rate for IMU data) of the IMU system is much higher than camera (e.g., video fv=30 frames per second, IMU fm=100 frames per second). Thus, a refinement step is performed to further improve accuracy of time bias up to ±1 IMU system frame.
With tdiff as initial temporal alignment, spatial alignment is performed as described with reference to
For each adjustment Δt, corresponding spatial alignment error is calculated and the temporal alignment with the smallest spatial alignment error is selected.
T
diff
=t
diff
+Δt
m
Δtm=argminΔt
where Err(.) denote the spatial alignment error measurement, v and MT denote the trajectories obtained from video and IMU system.
A computing device 2013 as a processor 2017, memory 2019, and a wireless module 2015. The computing device receives captured inertial measurement of the golf club swing from the inertial measurement unit 2001 attached to the golf club 2003, via the wireless module 2015 of the computing device 2013. Also, the computing device receives the captured video from the camera 2005 of the mobile device 2007, via the wireless module 2015 of the computing device 2013.
The computing device has a z axis alignment module 2021, a translation module 2023, an x y axis alignment module, a user assisted mode module 2027, a temporal alignment module 2029, and an overlay module 2031, each of which could be implemented in software executing on the processor 2017, hardware, firmware, or combination thereof. These modules implement functions described above with reference to
In a further embodiment, the computing device 2013 is integrated with the mobile device 2017, for example when the mobile device 2017 is a tablet that has a camera 2005, and inertial measurement unit 2009 and a display 2011. Many tablets, as is known, also have one or more processors and memory, a wireless module, user input devices, etc. In a still further embodiment, the camera 2005 is separate, and a display 2011 is coupled to or integrated with the computing device 2013. Video could be delivered from the camera 2005 to the computing device 2013 via a wired connection, or removal of memory from the camera 2005 and insertion of the memory into the computing device 2013, e.g., using a memory stick. In a yet further embodiment, the computing device 2013 is combined with the inertial measurement unit 2001 attached to the golf club 2003, so that the system needs only a mobile device 2007 with a camera 2005, or a separate camera 2005, and the inertial measurement unit with combined computing device 2013.
In an action 2101, a golf club swing is captured, using an inertial measurement unit attached to a golf club. In an action 2103, a video of the golf club swing is captured, using a camera. The camera could be part of a smart phone or a tablet or other mobile device, or a camera attached to a computing device, or a separate camera. These actions 2101, 2103 should be performed at the same time, so that the video captures the same golf club swing as is captured by the inertial measurement unit. The inertial measurement unit sends the captured golf club swing to a computing device, in various embodiments. The device with the camera sends the captured video of the golf club swing to a computing device, in various embodiments, or acts as the computing device in further embodiments.
In an action 2105, the z axis for the inertial measurement unit captured golf club swing and the captured video of the golf club swing are aligned. In an action 2107, translation of the inertial measurement unit captured golf club swing to the video of the golf club swing is determined. In an action 2109, x and y axes of the inertial measurement unit captured golf club swing and the captured video of the golf club swing are aligned. These actions 2105, 2107, 2109 can be performed by a computing device, e.g., by the processor of a computing device.
In an action 2111, a projected golf club swing trajectory is overlaid onto the video. In some embodiments, this uses time bias for temporal alignment. This action 2111 can be performed by the computing device. In an action 2113, the overlaid video is displayed. This action 2113 can be performed by the computing device, using a display attached to the computing device. In some embodiments, the computing device could be part of a mobile device, so that the mobile device is performing the actions 2105, 2107, 2109, 2111, 2113. Or, in some embodiments, the computing device sends the overlaid video to a mobile device, and the mobile device displays the overlaid video on a screen of the mobile device, e.g., on the screen of a tablet or a smart phone.
It should be appreciated that the methods described herein may be performed with a digital processing system, such as a conventional, general-purpose computer system. Special purpose computers, which are designed or programmed to perform only one function may be used in the alternative.
Display 2211 is in communication with CPU 2201, memory 2203, and mass storage device 2207, through bus 2205. Display 2211 is configured to display any visualization tools or reports associated with the system described herein. Input/output device 2209 is coupled to bus 2205 in order to communicate information in command selections to CPU 2201. It should be appreciated that data to and from external devices may be communicated through the input/output device 2209. CPU 2201 can be defined to execute the functionality described herein to enable the functionality described with reference to
Detailed illustrative embodiments are disclosed herein. However, specific functional details disclosed herein are merely representative for purposes of describing embodiments. Embodiments may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.
It should be understood that although the terms first, second, etc. may be used herein to describe various steps or calculations, these steps or calculations should not be limited by these terms. These terms are only used to distinguish one step or calculation from another. For example, a first calculation could be termed a second calculation, and, similarly, a second step could be termed a first step, without departing from the scope of this disclosure. As used herein, the term “and/or” and the “/” symbol includes any and all combinations of one or more of the associated listed items.
As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “includes”, and/or “including”, when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Therefore, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.
It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
With the above embodiments in mind, it should be understood that the embodiments might employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing. Any of the operations described herein that form part of the embodiments are useful machine operations. The embodiments also relate to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
A module, an application, a layer, an agent or other method-operable entity could be implemented as hardware, firmware, or a processor executing software, or combinations thereof. It should be appreciated that, where a software-based embodiment is disclosed herein, the software can be embodied in a physical machine such as a controller. For example, a controller could include a first module and a second module. A controller could be configured to perform various actions, e.g., of a method, an application, a layer or an agent.
The embodiments can also be embodied as computer readable code on a tangible non-transitory computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion. Embodiments described herein may be practiced with various computer system configurations including hand-held devices, tablets, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The embodiments can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.
Although the method operations were described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or the described operations may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing.
In various embodiments, one or more portions of the methods and mechanisms described herein may form part of a cloud-computing environment. In such embodiments, resources may be provided over the Internet as services according to one or more various models. Such models may include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). In IaaS, computer infrastructure is delivered as a service. In such a case, the computing equipment is generally owned and operated by the service provider. In the PaaS model, software tools and underlying equipment used by developers to develop software solutions may be provided as a service and hosted by the service provider. SaaS typically includes a service provider licensing software as a service on demand. The service provider may host the software, or may deploy the software to a customer for a given period of time. Numerous combinations of the above models are possible and are contemplated.
Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, the phrase “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs the task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. 112, sixth paragraph, for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configured to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
The foregoing description, for the purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the embodiments and its practical applications, to thereby enable others skilled in the art to best utilize the embodiments and various modifications as may be suited to the particular use contemplated. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
