Patent Application
20220171698
The present application relates to simulation and automated testing, and more particularly to systems and methods for simulating and automated testing of mobile video games using an automated agent.
A recurring problem in the field of videogames and 3D software applications is how to automatically test a product to find defects. While traditional form-based apps can benefit from User Interface (UI) Automation, 3D applications and video games usually cannot benefit from this approach, as their built-from-scratch UI is incompatible with prior automation packages for testing software.
Consequently, Quality Assurance (QA) analysts must manually exercise all the code paths of a videogame, trying to identify oddness in the behavior and/or images produced by the product. The nature of Mobile Application development further exacerbates this problem, due to the myriad different device manufacturers, chipset vendors and operating system versions in the market. A game that operates without errors on one device may fail on a similar device.
While prior approaches to automating mobile video games have been implemented, these have fallen short of automating the function of a QA analyst. For example, prior approaches have lacked the ability to simulate a player analyst who exercises different software code paths in search of potential errors. Nor is this a trivial technical problem to solve. Code paths leading to errors are not easily discovered for several reasons, such as the complexity of gaming code paths, the difficulty of accurately simulating human player behavior with an automated agent, and problems with distinguishing errors from acceptable outcomes from a human player's perspective. Because of these and other challenges, automated testing of 3D apps and videogames has been an open problem without a clear solution.
It would be desirable, therefore, to provide new methods and other new technologies able to simulate and automate testing of mobile video games to identify software defects and that overcome these and other limitations of the prior art.
This summary and the following detailed description should be interpreted as complementary parts of an integrated disclosure, which parts may include redundant subject matter and/or supplemental subject matter. An omission in either section does not indicate priority or relative importance of any element described in the integrated application. Differences between the sections may include supplemental disclosures of alternative embodiments, additional details, or alternative descriptions of identical embodiments using different terminology, as should be apparent from the respective disclosures.
In an aspect of the disclosure, a method evaluates performance of a video game by a computing device. In an aspect, the computing device is a mobile device and the video game is a mobile video game. In an aspect, the method automates a QA agent in mobile video games. In another aspect of the disclosure, an apparatus or local server automates continuous execution of a QA agent in mobile video games. In an aspect, the method may include three main components: (a) an on-device agent application, (b) a harness application and (c) an execution context. Generally, the harness application enables an automatic starting of a video game (e.g., “bootstrapping”) and deploying of an agent application that plays the game automatically in an execution context (e.g., a specific mobile phone or operating system). In some embodiments, the execution context may be a local device. In some embodiments, the execution context may be one or more servers, for example, continuous integration (CI) servers (e.g., a remote “device farm”) which includes hundreds of readily-available devices for testing purposes. The deployment of the “custom made” harness application allows the same tests to be “portable” between different execution contexts.
In an aspect, the method includes executing, by a processor of the computing device, the harness application for the video game. The method may deploy the harness application in remote devices, which are referred to as the computing devices. This advantageous deployment allows for scalable and automated testing of video games in mobile devices.
The harness application extracts variable values from a memory of the computing device and sends data based on the values to a data sink. The variable values include, for example, one or more predefined objectives of a video game play, and descriptions of an execution context. Data sent to the data sink may include test outcomes from games that are tested based on the predefined objectives of a video game play in an execution context. The data sink may be, or may include, a computer memory for storage or temporary use by an application.
Other data collected by the harness application may also include, for example, screenshots, current framerate, measure of CPU usage, measure of graphics processor usage, measure of memory usage, measure of power drain, or a measure of screen brightness. In an aspect, the data may be analyzed, and the analysis will help in developing future agent application for testing the video game or other video games.
In some embodiments, a data sink may be a computer-readable data storage medium or another computing medium. The data sink may be coupled to the computing device by a serial port or a network interface. In some embodiments, the data sink is hosted by a local apparatus or server. In some embodiments, the data sink may be hosted by one or more remote servers (e.g., at a device farm). The remote server may also host a data management application that causes the server to receive the data from the computing device and record the data in the data sink. The data management application may further cause the server to receive additional data like the data from multiple different computing devices contemporaneously performing the method.
Contemporaneously with executing the harness application, the method includes executing, by a processor of the computing device, an agent application for the video game that executes player commands for the video game based on a script and state of the computing device determined by the video game. In an aspect, the agent application may be encoded in a script, and is configured to simulate likely actions that real players may perform in a video game. In another aspect, agent application is encoded as a part of the video game. The agent builds an internal representation of the current state of the game and determines which actions may be played for each game state.
Contemporaneously with executing the harness application and the agent application, the method includes executing, by a processor of the computing device, the video game that accepts input generated by the agent application as player input. As the agent plays the video game it detects errors. For example, if the agent tries to perform an action that a player is not allowed to perform from a given screen according to its data file, it detects an error if the game allows the action to be completed. Conversely, if the game does not allow an action that is indicated in its data file as allowable, the agent may register an error. In an aspect, the agent application may detect and report errors to an interface or record for QA analysis.
In an aspect, the method includes continuing executing the harness application, the agent application, and the video game until achieving a predefined objective of video game play, for example, reaching a destination or completing a campaign. In an aspect, the agent application causes the computing device to read a data file that defines one or more game objectives. The data file may be preloaded in the mobile device contemporaneously with the deploying of the harness application. In addition or alternatively, the computing device may read the data file from an external data source through a communication interface, for example a network interface. The agent application then causes the computing device to generate simulate game-related user input based on the one or more game objectives.
In an aspect, the agent application may cause the computing device to generate the simulated user input at least in part by reacting to a detected state of the video game according to at least one predefined algorithm.
In an aspect, the method includes continuing executing the harness application, the agent application, and the video game indefinitely. In some embodiments, the method performs this execution from an apparatus or local server that automates continuous execution of the agent application in mobile video games.
The methods described herein provide an automated process for automatically evaluating performance of a video game by a computing device. Applications for the methods may include, for example, automatically testing of a video game in a mobile device. The method may include, for example, enabling a QA analyst to script and deploy simultaneously, on hundreds of devices, the on-device agent application that plays the video game automatically. The agent application simulates a real-life player and gathers execution statistics. In an exemplary application, the method may simulate high-level game-specific tasks such as “Go to the Campaign Map”, “Build a Superteam” or “Go to Battle”, as well as gather metrics like recording battery drain over time, frames-per-second performance and capture in-game screenshots, among others.
The foregoing methods may be implemented in any suitable programmable computing apparatus, by provided program instructions in a non-transitory computer-readable medium that, when executed by a computer processor, cause the apparatus to perform the described operations. The computer processor (or “processor”) may be local to the apparatus and user, located remotely, or may include a combination of local and remote processors. An apparatus may include a computer or set of connected computers that is used in audio-video or production or for output of audio-video or virtual or augmented reality content to one or more users. Other elements of the apparatus may include, for example, a user input device, which participate in the execution of the method.
To the accomplishment of the foregoing and related ends, one or more examples comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the annexed drawings set forth in detail certain illustrative aspects and are indicative of but a few of the various ways in which the principles of the examples may be employed. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings and the disclosed examples, which encompass all such aspects and their equivalents.
The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify like elements correspondingly throughout the specification and drawings.
Various aspects are now described with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that the various aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate describing these aspects and novel combinations of elements.
In an aspect, one or more processors may load the harness application inside the same process space as the rest of the video game. In some embodiments, security measures included in the one or more processors, in the mobile device and/or in the harness application may implement protection to prevent inadvertently unwanted software from being loaded with or in place of the harness application.
At computer process 104, the harness application determines the execution context it is operating in. In some embodiments, each mobile device has already contained the video game to be tested and an agent application. The on-device agent application tests the video game automatically by simulating actions of a real-life player and gathers execution statistics. The agent application may be programmed to receive and record high-level instructions from a user (e.g., QA engineer), and use an execution loop to perform the instructions. Using a lookup table or the like, the agent application distinguishes between successful execution of high-level instructions and errors. The agents compare clicks and other actions required to achieve a game result in a particular context to a standard for the context and flags any errors for human review.
At computer process 106, the harness application starts the execution of the video game. The video game then waits for input actions. In an aspect, at the computer process 108, the video game starts the execution of the on-device agent application which provides input actions to the video games to simulate a player's actions. The agent application monitors the testing, gathers statistics and screenshots if so directed. At computer process 110, the harness application waits for signal from the agent application. In an aspect where a single test run is configured, for example by the QA analyst or administrator, the agent application signals that testing of the video game has completed when, for example, all script instructions have been executed. At 112, once testing of the video game has completed, the harness application performs clean up tasks, for example storing test results, clearing and freeing memory and data usage. In an aspect where continuous test is configured, the agent application restarts from the first instruction.
One or more processors of the servers may enable automatic performance evaluation of a video game at one or more mobile devices with the execution of an on-device agent application that simulates actions of a player. In one aspect, the one or more processors determine, for example by retrieving from storage 304, the execution context 306 for each mobile device. Based on the execution context 306, the one or more processors load a corresponding harness application 308 to the mobile device. In an aspect, the one or more processors load an agent application script 310 built into a video game to the mobile device. The video game and the built-in agent application may be pre-loaded to the mobile device or may be loaded to the mobile device by the one or more processors contemporaneously to loading the harness application.
In an aspect, a monitor 310 is connected to the servers for displaying a live feed coming directly from the mobile device's screen. The monitor can also display full client and server logs. If any error is triggered within the video game, it may be displayed immediately on the monitor, including a full error trace and session log associated with it. For critical errors, the one or more processors may send an email, for example, to the technical leads of the test project. Quick access to error logs enables test and/or development engineers to identify problems immediately. The real-time monitoring and error reporting may allow engineers to fix minor issues without delay, and more complex issues can be reported to the QA analysts for further study and solution design.
In an aspect, each execution context structure links to a corresponding harness application. Knowing the execution context for a test case, the one or more processors can look up and load the corresponding harness application. For example, a single-device execution context entry 420 links to a harness application 422. Similarly, each execution context of a device farm structure 432 links to a corresponding harness application in a harness application table 440. In an aspect, the test case identifier 410 may link to a game identifier 450 which may link at least to a video game code 452. In an aspect, the test case identifier 410 may also link to an agent application script 460. As described herein, one or more processors may build the game code and agent application script in one package before loading into a mobile device for testing. In an aspect, the video game identifier can be null when the video game is pre-loaded in the device.
In an aspect, the agent application may be a Finite State Machine (FSM) automata whose purpose is to imitate a player. A QA analyst or engineer may configure the agent application directly according to the test to perform. The agent application differs from the traditional automated QA process, where simple UI Automation scripts are limited to simulating mouse clicks or button taps. With the agent application, the analyst can reason in terms of in-game actions a real-life player would perform. In an aspect, the agent application may include game-play functions, debug functions, performance profiling functions and generic functions. The following lists examples of functions of an agent application. In practice, game functions will vary depending on the game and version.
Game-Specific Gameplay Functions:
Game-Specific Debug Functions:
Performance Profiling Functions:
Generic Functions:
At 508, one or more processors at the device execution context start the execution of the harness application. In an aspect, at 510, the harness application extracts and gathers variable values, for example configuration variables, from the mobile device, creates the agent application and starts the game. In another aspect, the harness application may direct the game to create the agent application. At 512, the game may run the script which configures the agent application. In an aspect, the game passes the script as a code “handle” to the agent. In an aspect, the game may create a test strategy, for example as specified in the script. The agent application executes its code instructions, gathers statistics and screenshots if so directed. In an aspect, the one or more processors may contemporaneously send the statistics and screenshots to a display. At 514, the harness application monitors for errors and for signal of test completion if test is a single test run. In an aspect, when the test completes the one or more processors at the device execution context collects test results, performs clean-up functions and terminates execution of the harness application.
In an aspect, a server 710 may be constructed for each platform of interest, to enable concurrent testing of different platforms and to optimize the sever 710 for use with particular platforms. For example, it may be advantageous to build separate servers, one for testing iOS versions of games and one for Android versions.
At 912, the processor determines whether an object of interest has been found. If not, in an aspect, the processor returns to 908 to look for another object. In an aspect, the processor may also report ‘object not found’ (not shown). At 914, if an object of interest has been found, the processor determines whether a player action is allowed for the object in the current state of the game. In an aspect, if no action is allowed, the processor reports the condition at 920, then return to 908 to look for another object. If an action is allowed, the processor proceeds to 916 and performs the allowed action. At 918, the processor reports the result(s) of the action, stores any associated data, and updates the game (and agent application) state.
At 922, the processor determines whether the objective(s) for the test has(have) been achieved. If so, the processor reports the results and ends the test.
In an aspect, the on-device agent application can support capabilities that allow leveraging social features of a video game in order to simulate interactions between different players. This is especially valuable for the automated testing of video games with synchronous Player vs Player (PvP) interactions, resource trading and any other player-to-player interactions. Interactions like these, in a massive scale, are generally very cumbersome or outright impossible for a team of QA Analysts to test manually.
As illustrated in
The computing device 1000 may further comprise an electrical component 1004 for executing a harness application. The component 1004 may be, or may include, a means for said executing. Said means may include the processor 1010 coupled to the memory 1016, and to the network interface 1014, the processor executing an algorithm based on program instructions stored in the memory. Such algorithm may include a sequence of more detailed operations, for example, as described in connection with, for example, as described in connection with processes 102-104 and 504-508 of
The computing device 1000 may further comprise an electrical component 1006 for executing an agent application. The component 1006 may be, or may include, a means for said converting. Said means may include the processor 1010 coupled to the memory 1016, and to the network interface 1014, the processor executing an algorithm based on program instructions stored in the memory. Such algorithm may include a sequence of more detailed operations, for example, as described in connection with processes 108-110, 510-514 and method 900 of
The computing device 1000 may further comprise electrical components 1008 for executing a video game. The component 1008 may be, or may include, a means for said executing. Said means may include the processor 1010 coupled to the memory 1016, and to the network interface 1014, the processor executing an algorithm based on program instructions stored in the memory. Such algorithm may include a sequence of more detailed operations, for example, as described in connection with processes 106 and 510 of
As shown, the computing device 1000 may include a processor component 1010 having one or more processors, which may include a digital signal processor. The processor 1010, in such case, may be in operative communication with the modules 1002-1008 via a bus 1012 or other communication coupling, for example, a network. The processor 1010 may initiate and schedule the functions performed by electrical components 1002-1008.
In related aspects, the computing device 1000 may include a network interface module 1014 operable for communicating with a storage device, with servers, or other remote devices over a computer network. In further related aspects, the computing device 1000 may optionally include a module for storing information, such as, for example, a memory device/module 1016. The computer readable medium or the memory module 616 may be operatively coupled to the other components of the computing device 1000 via the bus 1012 or the like. The memory module 1016 may be adapted to store computer readable instructions and data for effecting the processes and behavior of the modules 1002-1008, and subcomponents thereof, or the processor 1010, or the methods described herein. The memory module 1016 may retain instructions for executing functions associated with the modules 1002-1008. While shown as being external to the memory 1016, it is to be understood that the modules 1002-1008 can exist within the memory 1016.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
As used in this application, the terms “component”, “module”, “system”, and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer or system of cooperating computers. By way of illustration, both an application running on a server and the server can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
In the foregoing description and in the figures, like elements are identified with like reference numerals. The use of “e.g.,” “etc.,” and “or” indicates non-exclusive alternatives without limitation, unless otherwise noted. The use of “including” or “include” means “including, but not limited to,” or “include, but not limited to,” unless otherwise noted.
As used herein, the term “and/or” placed between a first entity and a second entity means one of (1) the first entity, (2) the second entity, and (3) the first entity and the second entity. Multiple entities listed with “and/or” should be construed in the same manner, i.e., “one or more” of the entities so conjoined. Other entities may optionally be present other than the entities specifically identified by the “and/or” clause, whether related or unrelated to those entities specifically identified. Thus, as a non-limiting example, a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including entities other than B); in another embodiment, to B only (optionally including entities other than A); in yet another embodiment, to both A and B (optionally including other entities). These entities may refer to elements, actions, structures, steps, operations, values, and the like.
In many instances, entities are described herein as being coupled to other entities. It should be understood that the terms “coupled” and “connected” (or any of their forms) are used interchangeably herein and, in both cases, are generic to the direct coupling of two entities (without any non-negligible (e.g., parasitic) intervening entities) and the indirect coupling of two entities (with one or more non-negligible intervening entities). Where entities are shown as being directly coupled together or described as coupled together without description of any intervening entity, it should be understood that those entities can be indirectly coupled together as well unless the context clearly dictates otherwise. The definitions of the words or drawing elements described herein are meant to include not only the combination of elements which are literally set forth, but all equivalent structure, material or acts for performing substantially the same function in substantially the same way to obtain substantially the same result. In this sense it is therefore contemplated that an equivalent substitution of two or more elements may be made for any one of the elements described and its various embodiments or that a single element may be substituted for two or more elements in a claim.
Various aspects will be presented in terms of systems that may include several components, modules, and the like. It is to be understood and appreciated that the various systems may include additional components, modules, etc. and/or may not include all the components, modules, etc. discussed in connection with the figures. A combination of these approaches may also be used. The various aspects disclosed herein can be performed on electrical devices including devices that utilize touch screen display technologies and/or mouse-and-keyboard type interfaces. Examples of such devices include computers (desktop and mobile), smart phones, personal digital assistants (PDAs), and other electronic devices both wired and wireless.
In addition, the various illustrative logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
Operational aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
Furthermore, the one or more versions may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed aspects. Non-transitory computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD), BluRay™ . . . ), smart cards, solid-state devices (SSDs), and flash memory devices (e.g., card, stick). Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope of the disclosed aspects.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be clear to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
In view of the exemplary systems described supra, methodologies that may be implemented in accordance with the disclosed subject matter have been described with reference to several flow diagrams. While for purposes of simplicity of explanation, the methodologies are shown and described as a series of blocks, it is to be understood and appreciated that the claimed subject matter is not limited by the order of the blocks, as some blocks may occur in different orders and/or concurrently with other blocks from what is depicted and described herein. Moreover, not all illustrated blocks may be required to implement the methodologies described herein. Additionally, it should be further appreciated that the methodologies disclosed herein are capable of being stored on an article of manufacture to facilitate transporting and transferring such methodologies to computers.
The present application is a 371 of International Application Serial No. PCT/US2020/027838, filed Apr. 10, 2020, which claims priority to U.S. provisional patent application Ser. No. 62/832,541, filed Apr. 11, 2019, which are incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US20/27838 | 4/10/2020 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62832541 | Apr 2019 | US |
