Patent Application
20190163979
The present disclosure relates generally to managing electronic sensor data and more specifically, but not exclusively, to data processing systems for receiving and processing electronic data from accelerometers fitted on athletes, such as in gloves for combat sports.
A fan watching a sporting event often wonders how well they would perform a move they just saw performed by a professional athlete. For example, a baseball fan often wonders if they can hit a fast ball from their favorite professional pitcher. As another example, fans of combat sports or other competitive contact sports wonder how they would fare in the ring against their favorite professional athlete. Conventional systems do not enable the user to quantify the performance of the fan against that of a star athlete. For example, conventional systems do not enable the user to measure their own performance and track it against the performance of a star athlete. Among other challenges, conventional systems do not semantically associate video frames from a user with video frames of the star athlete. Conventional systems do not provide sensor data to evaluate the performance of the star athlete.
Furthermore, conventional video games and simulations of sporting events are either entirely fictitious or rely on generalized and derivative profiles measures. However, even generalized and derivative profiles measured to describe real world athletes and the ensuing action are also typically fictitious.
In view of the foregoing, a need exists for an improved athletic performance system and method for generating quantitative comparisons and predictive simulations from sensor data capturing the performance profile of professional athletes in an effort to overcome the aforementioned obstacles and deficiencies of conventional entertainment systems.
It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. It also should be noted that the figures are only intended to facilitate the description of the preferred embodiments. The figures do not illustrate every aspect of the described embodiments and do not limit the scope of the present disclosure.
Since currently-available entertainment systems are deficient because they fail to provide fans with a lifelike simulation or quantitative relative performance, an athletic performance system that enables a user to see how their own performance of a given sports move or routine compares with that of a professional athlete can prove desirable and provide a basis for a wide range of data metrics applications, such as the ability for fans to view a side by side video comparison of their performance against the performance of their favorite professional athlete. Furthermore, the athletic performance system can allow the predication of the sequence of events that follow an alternative action when one or more variables are changed through a simulation can also provide desirable and provide a basis for applications such as allowing fans and users to change the outcome of real matches as if the athlete themselves had made different choices in the execution of the game. These results can be achieved, according to one embodiment disclosed herein, by an athletic performance system 100.
Tuning to
In some embodiments, the central server 130 includes a communication system for receiving the sensor application data over a wireless network. The central server 130 can support one or more applications 140 and/or communicate with external services 150 described herein. Although described as a central server 130, those of ordinary skill in the art would understand that the central server 130 can be hosted on a cloud-service provider to support a platform for the performance management system 100.
Suitable wireless communication networks can include any category of conventional wireless communications, for example, radio, Wireless Fidelity (Wi-Fi), cellular, satellite, and broadcasting. Exemplary suitable wireless communication technologies include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband CDMA (W-CDMA), CDMA2000, IMT Single Carrier, Enhanced Data Rates for GSM Evolution (EDGE), Long-Term Evolution (LTE), LTE Advanced, Time-Division LTE (TD-LTE), High Performance Radio Local Area Network (HiperLAN), High Performance Radio Wide Area Network (HiperWAN), High Performance Radio Metropolitan Area Network (HiperMAN), Local Multipoint Distribution Service (LMDS), Worldwide Interoperability for Microwave Access (WiMAX), ZigBee, Bluetooth, Flash Orthogonal Frequency-Division Multiplexing (Flash-OFDM), High Capacity Spatial Division Multiple Access (HC-SDMA), iBurst, Universal Mobile Telecommunications System (UMTS), UMTS Time-Division Duplexing (UMTS-TDD), Evolved High Speed Packet Access (HSPA+), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Evolution-Data Optimized (EV-DO), Digital Enhanced Cordless Telecommunications (DECT) and others.
For example, with reference to
The fans then capture themselves attempting the same move, drill or routine using the camera and other sensors (e.g., internal motion and audio sensors) available to the mobile application 140 on their phone. The athletic performance system can receive the data for comparison at the platform 130 to then provide a side by comparison of the performance of the athlete and the fan, such as shown in
In some embodiments, the wireless communications between the subsystems of the athletic performance system 100 can be encrypted, as may be advantageous for secure applications. Suitable encryption methods include, but are not limited to, internet key exchange, Internet Protocol Security (IPsec), Kerberos, point-to-point protocol, transport layer security, SSID hiding, MAC ID filtering, Static IP addressing, 802.11 security, Wired Equivalent Privacy (WEP), Wi-Fi Protected Access (WPA), WPA2, Temporal Key Integrity Protocol (TKIP), Extensible Authentication Protocol (EAP), Lightweight Extensible Authentication Protocol (LEAP), Protected Extensible Authentication Protocol (PEAP), and the like. Encryption methods specifically designed for mobile platform management systems may also be suitable.
Thus, existing wireless technologies for use by current telecommunications endpoints can be readily adapted for use by the sensor applications and central server. For example, by outfitting each combat glove with a wireless card like those used for mobile phones, or other suitable wireless communications hardware, the combat gloves can easily be integrated into existing networks. Alternatively, and/or additionally, proprietary communications hardware can be used as needed.
In some embodiments, the sensor applications can include any number of sensors equipped on athletic equipment. Turning to
The sensor applications used by the fan and the professional can be the same type of sensor systems and/or include a combination of different (or unique) sensors. In IoT sensor systems, a wearable physical sensor is often mounted to the body of a person. This IoT sensor measures a physical aspect of the person, such as location, motion or biometric data. Location sensors can include wearable GPS sensors, UWB indoor sensors, and/or radio frequency identification (RFID) sensors. Motion sensors can include accelerometers, gyroscopes and magnetometers. Biometric sensors can include heartrate/heartrate variability, breathing, temperature, sweat, skin conductance, or electrically-determined muscle and brain activity.
Additionally and/or alternatively, video and audio data can be collected using cameras and microphones, respectively. Video and audio systems can be deployed for the purposes of collecting video and audio data during an event, or for general surveillance purposes, and used as inputs incidental to the event.
In the basketball example, the basketball player wears a UWB motion sensor that transmits its motion data to a wireless access point. The fan carries a smartphone in his pocket, running a mobile application that connects to accelerometers in the smartphone.
As previously discussed, data from the sensor applications are made available using a wireless communications network. This network might be a closed data network, virtual private network, open network, Internet, and/or IoT. The communications network might be established using one, individually, or a combination of more than one of wireless networks. Access points for wireless interfaces can be included in the communications system.
The communications network connects the data sources (e.g., sensor applications) to the data processing facilities (e.g., the central server 130). In the basketball example, the sensor on the basketball player communicates its data to a nearby access point, which communicates across a virtual private network over the Internet to the analytic processing system. The mobile applications that runs on the fan's smartphone communicates the sensor data through the cellular network over the Internet to the analytic processing system.
In some embodiments, as shown in
Specification
The fans then capture themselves attempting the same move, drill or routine using the camera and other sensors (e.g., inertial motion and/or audio sensors) available to the mobile application 140 on their phone. In other words, once an athlete triggers that a specific move is of interest, the performance measurement system can generate a video file and metadata relating that professional move. The video and metadata can include player/coach information, details about the moment (e.g., height of the jump, energy level of the athlete) and original video from a broadcast feed of the venue 110. A content management system stores the video and metadata, indexes the data, and enables content retrieval.
As discussed, the central server 130 can include the analytic processing system. The collected data are wirelessly communicated to the analytic processing system. The analytic processing system includes one or more analytic, machine learning, data mining and artificial intelligence methods. In some embodiments, machine learning methods include Hidden Markov Models, Random Forests, Unsupervised Clustering, Anomaly Detection and others. Analytics include peak detection, time-series analysis, computer vision, and data fusion. Data mining methods include correlation and statistical analysis. Artificial Intelligence methods include Deep Learning and Logic Modeling. Using these methods, selected characteristics of a time-series data source (e.g., video, audio, acceleration, rotation, position, and so on) are extracted for further processing. The selected characteristics are used to generate analytic signatures for a reference athlete and a user. These analytic signatures are compared to produce a similarity score. These methods produce a comparison between the actions of the two persons, as represented in the data collected by the sensor systems.
The processing system may be a single collocated assembly, or a distributed system interconnected by data communications networks. These may be implemented as physical hardware, or obtained virtually using cloud computing models.
In the basketball example, the analytic processing system is implemented as a cloud-based server cluster, residing in the datacenter of a cloud service provider.
The athletic performance system can determine the similarity between two complex sequences of sensor data values, producing a simple, low-complexity measure. This can be accomplished by representing the two action sequences as a time-series vector, and then performing a measure of the overlap of the two vectors. Stated in another way, a time series representing the reference athlete (e.g., professional athlete) can be compared with a time series representing the user to produce a single score, or collection of scores.
In some embodiments, the reference and the user data can include video data that represents the athlete performing a signature move and video data that represents the user attempting to replicate that same move. In each frame of the video, joint positions of each respective athlete can be determined using computer vision methods.
Determining joint positions can include any process described herein. For example, determining the joint positions from a video source can include segmentation and classification of image elements for each video frame. For a first video of a first user, the first user is distinguished from the background images. In other words, the first user is distinguished from the additional noise of the image. This identifies a collection of pixels in the image that are all associated with the person whose signature move is being analyzed.
Once the image is segmented, the image forms that represent different body parts are identified. This process includes supervised machine learning models that distinguish arms, legs, torso, head, facial features, and so on. The relationships between these identified elements are determined, so that the location of joints between body parts are returned. In some embodiments, the locations are determined within a reference frame internal to the video frame. Therefore, the relative location of different body parts to a specific frame reference can be used across video frames. By way of example, the vertical component of joint locations of a video frame can be used to ignore relative angle between the subject and the camera.
A time series of joint positions describes the movements of the athlete. This sequence of joint positions is used to compare to the video submitted by the user. For instance, the joint positions extracted from the video submitted by the user can be compared to those of the reference athlete to determine a similarity between the respective time-series of joint positions. This initial similarity measure describes the comparison of each frame in one video to a frame in a second video. This set of initial similarity measures can be aggregated across all frames in the video to produce a single comprehensive similarity score described herein.
This is accomplished by comparing the relative positions of the joints, between the reference and user. For instance, the frames from one video can be compared to frames in a second video to determine which reference frame is closest to which user frame. This produces an association between one video sequence and the other.
After this association is known, the locations of the body joints of the reference athlete in one frame can be compared to the locations of body parts of the user in the associated frame. This comparison can be made on the basis of separation of joints from each other, or the relative heights of the joints, relative to each other. It can also be done on the basis of the relative angles of the limbs, with respect to each other.
Accumulating metrics from one frame to the next yields a cumulative measure of the similarity between the reference video of the athlete and the video submitted by the user. This measure of similarity is then reported to the user as a similarity score. In some embodiments, the cumulative similarity score can be obtained using a summation of all individual similarity scores or a product of the individual scores.
In some embodiments, the comparison includes determining the product of the two time-series vectors, for each moment in time, and determining the sum of the resulting product. If the two time-series are highly correlated with each other, then this result will be large. If they are not, then the result is a small number. This value represents the measure of similarly.
As an example, the reference athlete can wear a motion sensor that measures acceleration and rotation. The time series of data from each of the three axes of reach sensor are recorded as the athlete performs a signature move. These data capture the significant aspects of the signature move, and characterize it using a simpler representation of the time series. For example, data derived from three axes of acceleration for an athlete can produce a single magnitude of acceleration. This advantageously simplifies the three values for an instant of time into a single value.
The time series data are compared in a similar manner as that of the sequence of joint positions discussed herein. Each represents the signature move in a simpler, lower-dimensional form.
First, an alignment of the time series is performed, in order to associate times in the reference data with times in the user-submitted data. In some embodiments, the alignment of the time series includes comparing individual time series data points or frames from both the first and second videos. While maintaining the sequential order of each of the sequence of video frames, the alignment includes identifying pairs of data points in the first video with similar data points from the second video to maximize a similarity score. In a preferred embodiment, the similarity score is maximized using a best fit of the sequential data points from the first video with that of the second video.
Next, the similarity of the associated data is determined using a correlation measure. The values of the acceleration and rotation are compared to determine the distance between them during each moment of time. This can be done by treating the three values of acceleration and the three values of rotation as a six-dimensional vector. The projection of one vector onto the other measures how similar the data are. This similarity is accumulated across the duration of the data. The result is a similarity measure that is reported to the user as the similarity score.
As shown in
In the basketball example, the motion data that represents the spinning dribble is directly compared between the basketball player and fan, to determine the degree of similarity between the player and fan. The motion data representing the dunk is also compared, to determine how high the fan jumped, relative to the player. These two comparisons are combined to produce a single numeric measure.
In some embodiments, the comparison measure is made available to one or both of the persons involved in the comparison. This may be delivered in one of several ways, including presentation within a mobile application, web page, email, text message, and/or push notification.
Advantageously, the athletic performance system provides a quantitative comparison between a professional athlete's and a fan's performance of a certain move, routine, or drill and can provide strategic value in the fields of entertainment, training and fitness. In entertainment, the ability to quickly—and without specialized equipment—produce such a side by side comparison creates a unique moment that is shareable in person and on social media. The system advantageously provides improved training and fitness for inspiring the fan to repeatedly perform routines like the professional athlete to see both how they match up in that moment and how much they have improved, (i.e., simulated the performance profile of the professional athlete).
The described embodiments are susceptible to various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the described embodiments are not to be limited to the particular forms or methods disclosed, but to the contrary, the present disclosure is to cover all modifications, equivalents, and alternatives.
This application is a non-provisional of, and claims the benefit of, U.S. Provisional Patent Application No. 62/592,337, filed Nov. 29, 2017, and U.S. Provisional Patent Application No. 62/592,357, filed Nov. 29, 2017, which applications are hereby incorporated herein by reference in their entirety for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62592337 | Nov 2017 | US | |
| 62592357 | Nov 2017 | US |
