Patent Application
20180211192
This document relates to data processing performed inside a gaming device such as an inflated ball.
Athletes, whether professional or amateur, accomplished or clumsy, go to great effort to improve their athletic performance. Many spend hours and hours on the field or court. Some hire personal coaches or go to camps that take great time and are very expensive. The drive to improve athletically and to win is deeply ingrained—so deeply that many people get joy simply from watching other people compete in athletics.
Much of the effort to watch and perform in sports is very subjective, with a coach watching a student's motion in taking a shot and providing feedback on it, or with spectators making similar judgments about players in games they attend.
This document describes systems and techniques that may be used to provide objective measures of athletic performance. In particular, described here are techniques by which the motion of a sporting device that is handled by an athlete, such as a ball in the form of a basketball, volleyball, or soccer ball, can be instrumented with motion sensors such as accelerometers, magnetometers, and gyroscopes, in the form of one or more inertial sensor packages. Such sensors may be located inside a ball, such as inside a shell or bladder for an inflated ball, and may move with the ball so as to sense motion imparted on the ball by one or more athletes.
Electronics that are in communication with the sensors and are also located inside the sporting device, such as in the form of a digital signal processor (DSP) and other electronics, may perform processing operations to turn the raw sensor data that does not have a meaning in the context of a particular sport, into derived data that is directed specifically to a particular sport. For example, raw accelerometer data in three axes may be converted into an indication of the g-force imparted on a ball when a player takes a shot, such as when a player dunks a basketball or kicks a soccer ball. Other raw data may be processed with certain stored assumptions about an athletic event, such as a stored a function about the release height of a basketball shot, so as to provide derived data that characterizes the shot using raw data from the sensor pack. As described in more detail below, other types of data may be generated with such an in ball processing system.
Such raw data, derived data, or both may then be provided to a computing system that is external to the ball such as by a wireless data communication formed between a wireless interface in the ball a computer outside the ball. For example, the data may be provided to a smartphone or tablet that is executing an application for causing an interface on the smartphone or tablet to communicate data with the electronics in the sporting device. Such an application may have been obtained from an online application store, such as the APPLE ITUNES STORE or the GOOGLE PLAY market, and may convert the data received from the ball into a graphical representation that may be readily viewed and interpreted by the athlete or by one or more other people, such as a coach, a referee, or a spectator of an athletic event. For example, data about the speed that a basketball shot was released, an angle of release of the basketball shot, and an amount of time between a player picking up a dribble and releasing the ball, or raising the ball and releasing the ball from a shot, may be displayed as text on a display of a smart phone or a tablet computer. Certain of the data may also be converted into a graphical form and such conversion may happen on the in-ball electronics, the computing device outside the ball, or in part on both. As one example, such electronics may compute an arc that a basketball shot took based on information received from the sensors in the ball, and the arc may be displayed in a graphical line on a background on the smartphone or tablet device. Such a displayed arc may be shown next to the best practices art that shows how the shot should have been aimed in a perfect world.
In this manner, the systems and techniques discussed here may permit for objective characterizations of the handling of the sporting device to be captured immediately and displayed in real time, for example in less than a second or 2 seconds of delay, in a visually pleasing manners on a variety of computing devices such as smart phones, tablets, heads-up displays in the form of Google glass head-mounted displays, and on other appropriate manners.
Other features and advantages will be apparent from the description and drawings, and from the claims.
In general, this disclosure relates to mechanisms by which data captured by a human manipulated game device by way of sensors mounted inside the game device may be processed by electronics that are also in the game device, from a form in which they are simple, raw motion data from an inertial sensor package, into a form that they are sports-specific data directed to representing particular actions in a particular type of sport, such as basketball, volleyball, soccer, or hockey. The transformations discussed here may be performed by a DSP with associated memory and may be performed on a single-chip DSP so that all mathematical manipulation of the data occurs on one chip. The memory may be provided on other chips or on the same chip, and the sensor package may be on other chips or on the same chip. All such chips may be mounted on a common motherboard inside the ball on multiple substrates. Other chips may also be provided in the ball, including wireless communication chips for providing communication with a computing device outside the ball.
The derived data that is produced by the in-ball electronics may be in a textual form transmitted as a binary code, where the text may include numbers that are human-understandable as relating to the particular sport. For example, the understandable numbers may include a number of times a basketball was bounced during a dribbling exercise, a g-force number indicating a force with which the basketball was bounced in terms of g-forces or other easily human-understandable formats, a number, in terms of degrees for an angle at which a ball was shot or a curve through which the ball curved during a shot, a number in terms of seconds or minutes between certain events that occurred in a sport. Raw data may also be transmitted outside of the ball instead of or in addition to the derived data just discussed. In certain implementations, the derived data may be data from across a relatively long period of sports activity, where such data summarizes the activities across that period. For example, the derived data may include numbers representing average force of dribbles over a time period, and other average values representing repetitive performance in a single representation.
Shown schematically inside ball 102 is processing unit 104, which may take a variety of forms and is shown in this example as a motherboard holding a number of connected chipsets and mounted solidly to the outer wall of the ball 102. The particular chipset shown in this example includes a sensor array 106, a microprocessor 108 which may take the form of a DSP, and a wireless interface 110.
The sensor array 106 may be a commercially available product that may include accelerometers, magnetometers, electronic gyroscopes, where each such instrument may sense forcer motion across three different axes, such as the X, Y, and Z axes. The sensor array 106 may include its own sensor data processor for converting, for example, analog signals into digital form and converting the digital signals into a form more representative and understandable by other electronics. For example, the sensor array 106 may produce outputs having digital values whose amplitude corresponds to the amplitude of acceleration sensed by the three different axes of accelerometers in the device. Similar signals may be provided for other parameters measured by the sensor array 106.
The processor 108 may take a variety of forms and may be connected to non-volatile memory storing instructions for performing preprocessing on the sensed data that is received from the sensor array 106. For example, the raw motion data just discussed may be converted into numbers representing particular sporting performance, such as a number of times that a basketball was bounced during a dribbling drill, a force for each bounce, averages for bounces across the multiple bounces that occurred during the drill and similar basketball-specific parameters.
The processor 108 may also have access to data memory and may store the raw motion data in such memory along with derived data from the processing just discussed. For example, a number of fields may be predefined by the software that executes on the processor, where a programmer identified those fields as being representative of a particular sport activity such as dribbling a basketball. Those fields may have corresponding fields for storage on devices 112 and 114 and thus may be wirelessly transferred from the ball 102 to the devices 112, 114. For example, a field may be provided for a number of times that a ball was bounced during a dribbling exercise, a number of times dribble was lost during the exercise, a force for each bounce in the dribbling, and an average force across all of the bounces in the dribbling, as just several examples.
The particular raw data is shown at 116 and may include X, Y, and Z axes for accelerometer data and angle or velocity data, among other things. In particular implementations, commercially available sensor arrays 106 may be obtained from a vendor and their outputs may be defined by such vendor and used to provide programming for the processor 108 to properly understand each incoming output and to process it accordingly into a form that is specific to the particular sport. Examples of such processed and derived data are shown at 114, and include, for example, for a dribbling drill, a number of dribbles in multiple fields including good dribbles, batted dribbles, or consecutive dribbles, and a number of times the athlete was forced to restart the dribbling exercise. Another field may indicate a speed of the dribbles such as a number of dribbles per minute, or a number of dribbles during the entire exercise. Yet another field may represent dribble intensity, which may be expressed in form of g-forces, average g-forces across all bounces or other manners.
As a second set of derived data, there are shown fields relating to a shot. Thus, for example, a user of device 112 may have chosen to perform an exercise that involves both dribbling and shooting, and upon the user making such a selection, device 112 may have provided a wireless communication to electronics in ball 102 causing them to gather such data upon sensing movement of the ball 100 to consistent with dribbling and shooting. The electronics in the ball may also determine when sensed motion indicates that the user has finished with the drill and upon making such a determination, may pass the data back out of the ball 102. In different implementations, data may be passed out of the ball 102 while the exercise is still occurring, or may be stored in volatile or nonvolatile memory in ball 102, processed when the exercise is completed, and then wirelessly transferred out to devices 112 or 114 upon the completion of such analysis.
In this example, the shot data is similar to that characterizing the dribbling data, and may include shot speed, shot arc, and shark shots in. In addition a shot launch height may be indicated and may have been computed in a variety of manners by electronics in ball 102. As a third instance of derived data, a shot distance may have been computed from various other measurements made by the ball electronics when the user took one or more shots as part of an exercise.
Starting with mobile device 204, there is shown a graphics GUI module 206 that may be part of an application that has been loaded on mobile device 204 and obtained by a user from a public application marketplace, for free or for a monetary fee. The graphics GUI module 206 may be responsible for converting textual data into a graphical form which may include visually-pleasing text in different fonts or in a drawn font, in addition to one or more graphics, animations, or videos that represent data that has been captured in a ball is providing the data to the mobile device 204.
A Bluetooth communication interface 210 in the mobile device 204 may include hardware and firmware and software that is intrinsically part of mobile device 204 and, in a familiar manner, converts wireless transmissions to data that can be used by one or more applications on the mobile device 204. A device bridge 208 performs a conversion that is needed for the graphics GUI module 206 to understand and work with the data that has been received by way of Bluetooth communication module 210.
Where the ball firmware 202 has performed extensive pre-processing of raw data received on a ball, the mobile device 204 may not need to perform substantial processing and may simply need to convert text such as in the form of representative athletic numbers, into graphics that improve the presentation of the data to a user so that such presentation is not merely color less single font graphics in a simple tabular form.
Referring now to the ball firmware, there is a Bluetooth communication module, which may take the form of commercially available hardware in the form of a chip programmed with firmware, software, or both for forming a Bluetooth connection with one or more devices in an area of the ball firmware 202 on a ball and for mediating a communication session in two directions with such devices.
A number of components are allowed to communicate using the Bluetooth communication module 212, such as a general control module 214. The general control module 214 may be programmed to execute on a processor of a device to manage communications between the device and a remote device, to provide necessary information to the remote device, and to coordinate the operation of other components of the ball firmware 202.
A drill manager 216 may be programmed to identify a drill that is to be sensed by the device, and to gather appropriate data from a sensor package in the form of an inertial sensor system for carrying out and characterizing the particular drill. A score engine 218 may be used to convert raw data obtained by the sensor package into particular scores in predefined fields or parameters that are directly related to a particular sport and activity that was performed with the ball with respect to such sport. A digital signal processor 220 is provided to perform traditional digital signal processing operations, including data manipulation with numerical data in the form of the raw data obtained by the sensor package. The sensor driver module 222 may also be accommodation of hardware, firmware, and software, and may be provided on a discrete chip or chips that are commercially available in a variety of forms.
A burn manager 224 may manage the storage of information in various forms of data on the device in one or more types of memory, while a battery manager 226 may manage the charging and discharging of a battery mounted within the ball that includes the electronics. For example, the battery may be charged by an inductive loop that receives energy from a loop outside the ball when the inductive loop is placed nearby the loop that is outside the ball. The battery manager 226 may control the amount of such energy that is provided to the battery, the speed with which the battery is charged and discharged, the level to which the battery is charged and discharged and other familiar battery management mechanisms. As with particular ones of the other modules, the battery manager 226 may be a commercially available chips or chips for providing low-power battery management services.
Fixed point DSP 304 is a digital signal processor that is a primary processor for the system. The DSP 304 may use firmware loaded on it to control all other components in the system 300 and process received raw sensor data into derived data in the form of key performance metrics for the particular sport that is being measured. DSP 304 may take the form, for example of a Texas instruments TMS320C5515AZCH10-C5515 digital signal processor available from Texas Instruments Inc.
SDRAM 310 is external RAM that expands high-speed memory that is available on the DSP 304. Such SDRAM 310 may be provided if RAM on DSP 304 is insufficient for the needs of the processing intended to be performed by DSP 304. Such RAM may be used to store program code, sensor data, or both. Generally, the SDRAM is a volatile memory and thus may be used to store various forms of the motion data during the processing operation. Where all of the necessary SDRAM is on DSP 304, the processing of the motion data may occur on such single chip without additional communications to other chips during the processing other than to provide input from the inertial sensor and output to one or more wireless communication structures. SDRAM 310 may take a variety of forms including the micron MT48H4M16LFB4-8 IT:H.
NOR flash 306 is external flash memory where firmware may be stored in a non-volatile form. Such memory may store both bootloader firmware as well as main program firmware that may be loaded onto DSP 304 for performing the data analysis described above and below. NOR flash 306 may take a variety of forms, such as the Micron M28W320FCB70ZB6E.
Inertial sensor 308 with its included sensor data processor may be one of three integrated circuits in system 300 that sent inertial motion from this component, acceleration along three axes, angle or velocity about three axes, and three axes magnetometer data can be collected for providing to the DSP 304. The inertial sensor 308 may also perform calibration an inertial motion calculations such as estimating object orientation and object no motion event. In certain implementations, the inertial sensor 308 may be used for raw acceleration data along three axes an angular velocity about three axes. Such a device may take the form of a InvenSense MPU-9150.
Not specifically shown here, but which may be provided on the backside of motherboard 302 or in other locations are a wireless charging integrated circuit which may manage the wireless battery charging, including inductive charging, from a base station. Such a device may take the form of a Texas instrumentsBQ51014YFPT. In addition, a Bluetooth radio may be provided which is capable of Bluetooth communications using Bluetooth classic or Bluetooth low energy communication standards such a device may take the form of a Texas instruments CC2564RVMR. In appropriate circumstances, a separate magnetometer may be provided providing three axes data, and may take the form, for example, of a Honeywell HMC5983. Moreover, a three axis accelerometer integrated circuit package may be provided that may sense acceleration along three axes. Such a device may also be configured to sensed simple motion event such as impact freefall, and double tap. Such component may also be provided to wake up the system, such as by sensing that a ball that has been nonmoving for a long. Has suddenly been bounced a certain number of times or otherwise manipulated in a predetermined manner by an athlete. Such a component may take the form of a ST Microelectronics LIS 3DH. Each of the sensors contains internal analog to digital converters that produce 16-bit integer outputs to be processed by other components in the system.
The DSP 304 serves as the master device in the system 300. Upon startup, such as occurring by sensing a predetermined motion of the sporting device by one of the sensors, firmware is loaded from the NOR flash 306 into the RAM of the DSP 304. The DSP 304 then configures the communication between all other integrated circuits on the board 302. The DSP 304 controls the sensors, such as inertial sensor 308, to collect synchronized accelerometer, angle or rate gyro.
The system 300 connects over a Bluetooth link to a remote mobile device, such as a device operating the IOS or android operating system. Data analysis begins when such a mobile device communicates a packet telling the system 300 to start an activity. The payload of such a packet may pass configuration parameters to the system 300. The command may signal the ball or the electronic shown here that the mobile device is expecting to receive notification about the balls current activity. Once the command is received and the DSP 304 begins collecting and processing sensor data, such data may be passed back to the requesting mobile device. The DSP 304 may collect the data in a variety of forms, such as at 1024 Hz from the accelerometer and Angel or rate gyroscopes, and synchronize such data with 200 Hz data from the magnetometer.
The raw data from the sensors, such as inertial sensor 308 may be stored in a circular buffer in the DSP 304 memory, or in SDRAM 310 when such additional memory is required. The circular buffer may hold a certain period of data, such as one, two, three, four, or five seconds of data. The raw sensor data may be initially filtered and then analyzed continuously. The filter data buffer may be passed through complex logic and statistical correlations to identify key timing events, such as contact of a basketball with the floor, with a user's hands, or other events that represent, for example, the start or end of a particular process as part of an athletic action, or the occurrence in time of a particular athletic action. The derived data from such analysis may be sent to the remote mobile device in near real-time, such as within a single digit number of seconds of the data being captured.
The activity notification packets that are transferred from the system 300 back to the mobile device are arranged to require little to no processing on the mobile device in certain implementations. In particular, the values of data that are provided to the mobile device may be left unchanged by the mobile device between their receipt and their display. The mobile device may instead be asked only to provide formatting for such display. For example, the mobile device may turn a series of numbers into a graphical display, such as a line graph or other such display, though the mobile device would not manipulate the numbers into another and different numerical representation.
At step 408, an end request is provided and responded to, and indicates that the device application would like the ball to stop performing the data capture and analysis that is been performing. At step 410, a get activity data command is provided in your responses provided in return. For example certain data may be passed off the ball while it is still being manipulated, and the get activity data command 410 may cause additional stored data to be passed from the ball to the device application.
At step 420, a set real-time clock command may be provided to the in-ball electronics and responded to. The in-ball electronics may then set their clock to be synchronized with a clock in the device application to ensure a tight synchronization between the two clocks during subsequent operations and communications between the two. At step 426, a get status request is provided and responded to by which the in-ball electronics may report the status of the ball, where such status may include indicating that the ball is at rest, the ball is undergoing motion, the ball electronics are lacking in sufficient electrical power, and other status information a variety of forms.
At step 500, a ball and external computer are automatically wirelessly connected. Such connection may take a familiar form such as a handshake according to a known and common protocol, such as Bluetooth. At step 503, an activity with the ball starts. Such activity may be initiated by an operator of the external computer interacting with an application on the external computer so as to cause an initiation command to be passed to electronics in the ball. The ball may then load firmware and begin monitoring data being generated by various sensors in the ball in a looping and continuous manner, and processing such raw data to produce derived data like that discussed above.
At box 504, a request for the processed data is provided from the external computer to the in-ball electronics, and at box 506, it is responded to. Such request and response may occur repeatedly and essentially continuously, or the in-ball electronics can provide data continuously for a long period in response to a single request and continue until a “stop” request is received. At box 508, the activity ends. For example, the in-ball electronics may identify from the data that sports-related motion of the ball has stopped (e.g., the ball has been placed on a cart for half-time), or the computer may provide a command to cause the in-ball electronics to stop collecting and processing data.
At box 510, activity data is obtained from the in-ball electronics to the external computer. Such recovery of data may occur, for example, when data was not obtained while the ball was being handled or may be obtained in addition. Also, a sub-set of the obtained raw data or derived data may be transmitted while the activities are ongoing, and the rest may be transmitted after the activity is determined to have stopped.
The particular techniques described here may be assisted by the use of one or more computers, such as wearable computers (e.g., in the form of glasses with a heads-up display), servers, desktops, smartphones, and tablets. The computing portions of such devices are shown generally in
The system 600 may be implemented in various forms of digital computers, including computerized laptops, personal digital assistants, tablets, and other appropriate computers. Additionally the system can include portable storage media, such as, Universal Serial Bus (USB) flash drives. For example, the USB flash drives may store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.
The system 600 includes a processor 610, a memory 620, a storage device 630, and an input/output device 640. Each of the components 610, 620, 630, and 640 are interconnected using a system bus 650. The processor 610 is capable of processing instructions for execution within the system 600. The processor may be designed using any of a number of architectures. For example, the processor 610 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor.
In one implementation, the processor 610 is a single-threaded processor. In another implementation, the processor 610 is a multi-threaded processor. The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630 to display graphical information for a user interface on the input/output device 640.
The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit.
The storage device 630 is capable of providing mass storage for the system 600. In one implementation, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.
The input/output device 640 provides input/output operations for the system 600. In one implementation, the input/output device 640 includes a keyboard and/or pointing device. In another implementation, the input/output device 640 includes a display unit for displaying graphical user interfaces.
The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).
To provide for interaction with a user, the features can be implemented on a computer having an LCD (liquid crystal display) or LED display for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer.
The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.
The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the described one. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
Many other implementations other than those described may be employed, and may be encompassed by the following claims.
This application claims the benefit of priority under 35 U.S.C. 119 to U.S. Application No. 61/946,702 filed Feb. 28, 2014. This application is a continuation of U.S. application Ser. No. 14/631,109 filed Feb. 25, 2015, and which is currently pending as of the filing of this application.
| Number | Date | Country | |
|---|---|---|---|
| 61946702 | Feb 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14631109 | Feb 2015 | US |
| Child | 15936218 | US |
