Patent Application
20210154584
This disclosure is directed to systems and methods for processing video game data, and in particular, to processing video game data on a computing device based on outputs from a game controller.
In video editing application, to easily identify and trim video gameplay video, typically snapshots of the video frames are displayed in a marquee interface. Generally, this marquee interface is displayed directly below a timeline axis such that a particular position in the timeline corresponds directly to a frame of video. However, these small previews are not particularly useful to locate specific moments in a video gameplay due to the coarse nature of how the frames are displayed. As the timeline or video grows in duration, the conventional snapshot approach becomes less and less useful, requiring a very coarse representation or a scrolling mechanism. Furthermore, a video alone only contains audio and video data, and it can be very difficult to extract meaningful data without a huge amount of video computation.
Embodiments of the disclosure address these and other deficiencies of the prior art.
Aspects, features and advantages of embodiments of the present disclosure will become apparent from the following description of embodiments in reference to the appended drawings in which:
Embodiments of the disclosure provide a device and method for extracting gameplay information to identify important points or regions in video gameplay footage. As will be discussed in more detail below, one or more of game controller inputs, sensor inputs, and audio data may be analyzed to determine the interesting points or regions in the video gameplay footage, which may be done by outputting time series data. The time series data may include aligning the video gameplay footage with controller inputs and marking points of interest in the time series data. The time series data may be displayed concurrently to a user with video viewing and editing functions.
The game controller 102 includes one or more user controls 106, such as, but not limited to, buttons, switches, joysticks, etc., which a user may use when playing a video game displayed on the computing device 100 or otherwise interacting with the computing device 100. For example, the game controller 102 may be a Human Interface Device (HID), such as a keyboard, mouse, or any input device having a combination of buttons, switches, and/or joysticks to interact with a video game. The output of the user controls 106 are sent to the controller interface 104 through one or more processors 108. The one or more processors 108 may sample the user controls 106 at a high rate and output those high rate samples to the computing device 100 to control various aspects of a game being displayed on the computing device 100. As will be discussed in more detail below, the one or more processors 108 may also packetize the user controls 106 for output to a controller unit 112 on the computing device 100 to assist with identifying points of interest in a video game being played by the user. The output from game controller 102 may then be received by an operating system 110 of the computing device 100.
The operating system 110 may pass the user controls to the game unit 114 and/or the controller unit 112. The game unit 114 may stream from a cloud or other connected device the video game or store the video game directly on the computing device 100. The operating system 110 can coordinate the sending and receiving of data with the game controller 102 and the controller unit 112, so that the controller unit 112 can process the incoming user control data. Further, as will be understood by one skilled in the art, the computing device 100 may include a number of connected, but separate, components that cooperate in conjunction with each other to achieve the operations discussed below. For example, the display, the game unit 114, and/or the controller unit 112 may be located on different devices and/or different processors.
During operation, the game unit 114 can cause a video game to display on a display 118 to a user, which can be played by the user through the controller 102 using the user controls 106. The high sampled rate user controls from the game controller 102 are sent to the game unit 114 for operation of the game. However, a packetized version of the user controls are also sent to the controller 112 to identify points of interest in the video game being displayed by the game unit 114. The game unit 114 can be operating in the foreground of the computing device 100 while the controller unit 112 is operating in the background. That is, the user will be interacting directly with the game controller 102 and the game displayed by the game unit 114, while the controller unit 112 is receiving and processing data in the background.
The operating system 110 may include a recording unit 116 or any other component that captures the video game data displayed by the game 114. This captured video can then be sent to the controller unit 112 for processing either in real-time or near real-time, or once the game has been completed by the user and recording has ceased. In some embodiments, the recording unit 116 may also capture not just the video of the game being played, but also the audio of the game being played.
When the controller unit 112 has received both the user outputs from the controller 102 and the recorded video game from the recording unit 116, the controller unit 112 can process the user outputs to generate time-series data, which as will be discussed in more detail below, can be used to highlight points of interest in the recorded video game to assist a user with editing the recorded video game. In some embodiments, the controller unit 112 may be located on another device, such as a personal computer and the controller unit 112 may receive the recording from the recording unit 116, as well as the user controls 106 through the interface 104.
Although
As another example, the game unit 114 may be stored locally on the computing device 100, and may be presented on a display 118 either attached to, such as a television or computer monitor, or incorporated into the computing device 100, such as a screen of a mobile device, laptop, or tablet, for example. In some embodiments, the controller unit 112 may be stored locally as well on the computing device 100, as illustrated in
Initially in operation 200, the one or more processors 108 sample the output from the user controls 106 at a high rate, such as 120 Hertz (Hz). In operation 202, the sampled outputs can be aggregated into longer duration frames by the one or more processors 108. That is, data may be collected for a period of time and aggregated into a frame. The period of time may be any period of time, such as, but not limited to, a quarter of a second or half a second. During the frame aggregation of operation 202, the one or more processors 108 also packet the aggregated frames to be sent to the interface 104 in operation 206.
The information is transmitted from the interface 104 to the controller unit 112 in operation 206. In operation 206, the controller unit 112 can instruct a memory (not shown) to store the packets from the game controller 102. In some embodiments, in operation 208, the controller unit 112 can begin processing the data as soon as it is received. In other embodiments, the processing may be delayed until a larger set of data is received from the interface 104. The processing in operation 208 runs continuously until a recording of the video game has ceased and the last of the data from the game controller 102 is received.
Processing in operation 208 includes determining periods of interest in the video game based on the received data from the game controller 100. For example, if higher game controller 102 activity is identified by the controller unit 112 in a portion of the data, then this data can be identified as a period of interest.
In some embodiments, game audio data may also be collected, as illustrated in operation 210. The audio data may be collected, for example, by the recording unit 116. The controller unit 112 may then in operation 208 process the audio data to identify periods of interest. For example, loud portions of the video game may be identified as a period of interest or, loud portions of the video game may identified as a period of interest if the user controls during that time period are above a certain threshold of activity. For example, although the user control 106 for a certain period alone may not equate to a point of interest, if the audio from the game at this time period is louder than other points, the controller unit 112 may label this section as a period of interest.
In operation 212, the controller unit 112 can generate time series data, which includes aligning the recorded video game from the recording unit 116 and the inputs from the game controller 100. Periods of interested identified by the controller 112 can then be aligned with the recorded data to assist a user in identifying interesting or important moments in the game. In some embodiments, a graph of the user control data can be generated by the controller unit 112. The graph may be, for example, a bar plot or a waveform, to indicate the amount of activity determined by the controller unit 112 for the user controls 106.
Although not illustrated, in some embodiments, a sensor may be provided on the game controller 102 to provide additional data and/or to provide the output for determining points of interest in the game. For example, the sensor may be a motion sensor, such as an accelerometer, to sense movement of the controller 102. The sensor data may be used to indicate when the game is being played versus navigating controls on the computing device 100, for example. In other embodiments, the sensor data may also be used by the controller unit 112 to determine points of interest in the game. For example, if the game controller 102 itself is moving a lot, it may indicate that the user is playing an interesting portion of the game. In some embodiments, the game controller 102 may be worn by a user, and the amount of motion sensed by a sensor in the game controller 102 may be used as the output for determining the points of interest in the game. In such embodiments, the user inputs 106 are not used to determine the amount of activity, but rather the sensor data is used to determine the amount of activity.
The activity is output as y in
The switches, buttons, and trigger vectors x2 and a switches, buttons and trigger weight w2 are combined in dot product block 304 and added to a minimum activation bias b in the summer 306. The output of the dot product block 302 and the summer 306 are then added together through summer 308 to output the amount of level activity y. However, as will be understood by one skilled in the art, this is just one example of how the activity may be determined from the switches, buttons, triggers, and joystick outputs. Further processing may be provided to the dot product output 302 or the summer output 306 to more accurately determine the amount of activity.
The final vector for the button and the joystick are then combined in the summer 314 to determine the output y, which is recorded over time and may be normalized when all of the data is received. Output y can then be graphed by the controller unit 112 as a waveform or bar graph, for example, to compare to the recorded video game. An activity threshold may be set to determine the points of interest. For example, if the output data is above the activity threshold, that point in the video game is marked as a point of interest. In some embodiments, the activity threshold may be a set threshold, or the activity threshold may be determined based on the amount of activity detected during the recorded video game. For example, a recorded game with less activity overall may have a lower threshold in some embodiments.
The graphical user interface 400 may include a video preview window 402 to display the recorded video game. A timeline bar 404 is also displayed and may contain a number of frames 406 of the recorded video game. In some embodiments, the timeline bar 404 may also be used to trim the recorded video game.
In some embodiments, the graphical user interface 400 may also include an activity graph 406. The activity graph 406 is time-aligned with the timeline bar 404 and can display the amount of activity in certain areas. In some embodiments, as illustrated in
In some embodiments, as illustrated in
As discussed above, embodiments of the disclosure allow a user to quickly identify points of the game which may be particularly interesting based on the amount of activity captures on the game controller 102. Generally, the more activity the game controller 102 is receiving, the more interesting that point of the game will be to a user. Embodiments of the disclosure allow a user to quickly discern which areas of the recorded video game may be of interest, without having to scroll or watch through the entire recorded video to identify those areas.
Aspects of the disclosure may operate on particularly created hardware, firmware, digital signal processors, or on a specially programmed computer including a processor operating according to programmed instructions. The terms controller or processor as used herein are intended to include microprocessors, microcomputers, Application Specific Integrated Circuits (ASICs), and dedicated hardware controllers. One or more aspects of the disclosure may be embodied in computer-usable data and computer-executable instructions, such as in one or more program modules, executed by one or more computers (including monitoring modules), or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device. The computer executable instructions may be stored on a computer readable storage medium such as a hard disk, optical disk, removable storage media, solid state memory, Random Access Memory (RAM), etc. As will be appreciated by one of skill in the art, the functionality of the program modules may be combined or distributed as desired in various aspects. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, FPGA, and the like. Particular data structures may be used to more effectively implement one or more aspects of the disclosure, and such data structures are contemplated within the scope of computer executable instructions and computer-usable data described herein.
The disclosed aspects may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed aspects may also be implemented as instructions carried by or stored on one or more or computer-readable storage media, which may be read and executed by one or more processors. Such instructions may be referred to as a computer program product. Computer-readable media, as discussed herein, means any media that can be accessed by a computing device. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media.
Computer storage media means any medium that can be used to store computer-readable information. By way of example, and not limitation, computer storage media may include RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Video Disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, and any other volatile or nonvolatile, removable or non-removable media implemented in any technology. Computer storage media excludes signals per se and transitory forms of signal transmission.
Communication media means any media that can be used for the communication of computer-readable information. By way of example, and not limitation, communication media may include coaxial cables, fiber-optic cables, air, or any other media suitable for the communication of electrical, optical, Radio Frequency (RF), infrared, acoustic or other types of signals.
The previously described versions of the disclosed subject matter have many advantages that were either described or would be apparent to a person of ordinary skill. Even so, these advantages or features are not required in all versions of the disclosed apparatus, systems, or methods.
Additionally, this written description makes reference to particular features. It is to be understood that the disclosure in this specification includes all possible combinations of those particular features. Where a particular feature is disclosed in the context of a particular aspect or example, that feature can also be used, to the extent possible, in the context of other aspects and examples.
Also, when reference is made in this application to a method having two or more defined steps or operations, the defined steps or operations can be carried out in any order or simultaneously, unless the context excludes those possibilities.
Although specific examples of the invention have been illustrated and described for purposes of illustration, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, the invention should not be limited except as by the appended claims.
