Patent Application
20240375010
Embodiments of the invention relate generally to systems and methods of reinforcement learning. More particularly, embodiments of the invention relate to methods and systems for universal value function approximators (UVFA)-like goals based on compositional reward functions parameterized by their components' weights.
The following background information may present examples of specific aspects of the prior art (e.g., without limitation, approaches, facts, or common wisdom) that, while expected to be helpful to further educate the reader as to additional aspects of the prior art, is not to be construed as limiting the present invention, or any embodiments thereof, to anything stated or implied therein or inferred thereupon.
In the field of Reinforcement Learning, value functions Vπ (s) are used to model the expected future reward for an agent starting in a state s and following a policy π. This value function is then either used by the agent to directly decide on an action to take or to inform and stabilize the learning process of a separate policy function in an actor-critic setting.
Universal value function approximators (UVFA), Vπ (s,g), are an extension of value functions that are additionally conditioned on a goal g, i.e., they estimate the future rewards starting from state s with the reward function depending on the active goal g. This allows a UVFA-based agent to learn how to best behave under multiple goals and potentially generalize to unseen goals. Exemplary goals for UVFA include a discrete set of goal states (e.g., 2D goal positions in a grid world with the agent rewarded for reaching the active goal position); or a vector representation of arbitrary pseudo-reward functions.
In view of the foregoing, there is a need for improved formulation for UVFA-like goals based on compositional reward functions parameterized by their components' weights
Aspects of the present invention provide an improved formulation for UVFA-like goals based on compositional reward functions parameterized by their components' weights. Additionally, aspects of the present invention provide a set of reward components for the domain of autonomous racing games that, when combined with the improved UVFA formulation, allows training a single racing agent that generalizes over continuous behaviors in multiple dimensions. This can be used by game designers to tune the skill and personality of a trained agent.
Embodiments of the present invention provide a method and a non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out the method of training an artificial intelligent agent that generalizes over continuous behaviors in multiple dimensions, wherein the method comprises defining a reward function based on a state and an action as a linear combination of a plurality of component reward functions and a weight for each of the plurality of component reward functions; sampling multiple dimensions of the weight for each of the plurality of component reward functions from a continuous distribution between a maximum weight and a minimum weight; and training a single policy of the artificial intelligent agent over a continuous goal space including a plurality of parameterized reward functions represented by the continuous distribution of the weight for each of the plurality of component reward functions.
Embodiments of the present invention provide a method for providing an artificial intelligent agent in a racing game that is tunable to one or more skill components and/or one or more personality components comprising defining a reward function based on a state and an action as a linear combination of a plurality of component reward functions and a weight for each of the plurality of component reward functions; sampling multiple dimensions of the weight for each of the plurality of component reward functions from a continuous distribution between a maximum weight and a minimum weight; training a single policy of the artificial intelligent agent over a continuous goal space including a plurality of parameterized reward functions represented by the continuous distribution of the weight for each of the plurality of component reward functions, wherein the plurality of component reward functions include a base reward, motivating the artificial intelligent agent to finish a race in a minimal time, and one or more additional component reward functions, providing the one or more skill components and/or the one or more personality components.
These and other features, aspects and advantages of the present invention will become better understood with reference to the following drawings, description and claims.
Some embodiments of the present invention are illustrated as an example and are not limited by the figures of the accompanying drawings, in which like references may indicate similar elements.
Unless otherwise indicated, the figures are not necessarily drawn to scale.
The invention and its various embodiments can now be better understood by turning to the following detailed description wherein illustrated embodiments are described. It is to be expressly understood that the illustrated embodiments are set forth as examples and not by way of limitations on the invention as ultimately defined in the claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well as the singular forms, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.
Unless otherwise defined, all terms (including technical and scientific terms)
used herein have the same meaning as commonly understood by one having ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
In describing the invention, it will be understood that a number of techniques and steps are disclosed. Each of these has individual benefit and each can also be used in conjunction with one or more, or in some cases all, of the other disclosed techniques. Accordingly, for the sake of clarity, this description will refrain from repeating every possible combination of the individual steps in an unnecessary fashion. Nevertheless, the specification and claims should be read with the understanding that such combinations are entirely within the scope of the invention and the claims.
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without these specific details.
The present disclosure is to be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments illustrated by the figures or description below.
A “computer” or “computing device” may refer to one or more apparatus and/or one or more systems that are capable of accepting a structured input, processing the structured input according to prescribed rules, and producing results of the processing as output. Examples of a computer or computing device may include: a computer; a stationary and/or portable computer; a computer having a single processor, multiple processors, or multi-core processors, which may operate in parallel and/or not in parallel; a supercomputer; a mainframe; a super mini-computer; a mini-computer; a workstation; a micro-computer; a server; a client; an interactive television; a web appliance; a telecommunications device with internet access; a hybrid combination of a computer and an interactive television; a portable computer; a tablet personal computer (PC); a personal digital assistant (PDA); a portable telephone; application-specific hardware to emulate a computer and/or software, such as, for example, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific instruction-set processor (ASIP), a chip, chips, a system on a chip, or a chip set; a data acquisition device; an optical computer; a quantum computer; a biological computer; and generally, an apparatus that may accept data, process data according to one or more stored software programs, generate results, and typically include input, output, storage, arithmetic, logic, and control units.
“Software” or “application” may refer to prescribed rules to operate a computer. Examples of software or applications may include code segments in one or more computer-readable languages; graphical and or/textual instructions; applets; pre-compiled code; interpreted code; compiled code; and computer programs.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
Further, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously.
It will be readily apparent that the various methods and algorithms described herein may be implemented by, e.g., appropriately programmed general purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) will receive instructions from a memory or like device, and execute those instructions, thereby performing a process defined by those instructions. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of known media.
The term “computer-readable medium” as used herein refers to any medium that participates in providing data (e.g., instructions) which may be read by a computer, a processor or a like device. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves and electromagnetic emissions, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
Various forms of computer readable media may be involved in carrying sequences of instructions to a processor. For example, sequences of instruction (i) may be delivered from RAM to a processor, (ii) may be carried over a wireless transmission medium, and/or (iii) may be formatted according to numerous formats, standards or protocols, such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G or the like.
Embodiments of the present invention may include apparatuses for performing the operations disclosed herein. An apparatus may be specially constructed for the desired purposes, or it may comprise a device selectively activated or reconfigured by a program stored in the device.
Unless specifically stated otherwise, and as may be apparent from the following description and claims, it should be appreciated that throughout the specification descriptions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulate and/or transform data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices.
In a similar manner, the term “processor” may refer to any device or portion of a device that processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory or may be communicated to an external device so as to cause physical changes or actuation of the external device.
As is well known to those skilled in the art, many careful considerations and compromises typically must be made when designing for the optimal configuration of a commercial implementation of any method or system, and in particular, the embodiments of the present invention. A commercial implementation in accordance with the spirit and teachings of the present invention may be configured according to the needs of the particular application, whereby any aspect(s), feature(s), function(s), result(s), component(s), approach(es), or step(s) of the teachings related to any described embodiment of the present invention may be suitably omitted, included, adapted, mixed and matched, or improved and/or optimized by those skilled in the art, using their average skills and known techniques, to achieve the desired implementation that addresses the needs of the particular application.
Broadly, embodiments of the present invention provide systems and methods to develop artificial intelligence (AI) policies for artificial agents for various domains, including gaming domains, such as racing games. The behavior of such AI agents can be user selected at run time by selecting parameters for a plurality of different factors. A single policy can be trained to handle the user selection of parameters across a predetermined range for each component. The agents can be trained across a number of weights within the desired range for each component. These weights determine how much of a reward portion for each component should be considered by the agent during training. Thus, an improved formulation can be realized for UVFA-like goals based on compositional reward functions parameterized by their components' weights. Additionally, a set of reward components has been determined for the domain of autonomous racing games that, when combined with the improved UVFA formulation, allows training a single racing agent that generalizes over continuous behaviors in multiple dimensions. This can be used by game designers to tune the skill and personality of a trained agent.
When AI policies are developed for different games, different players are playing with different skill levels. Thus, when playing against an AI agent, a player often desires this agent to play at a skill level at least equal to that of the player. Aspects of the present invention provide reinforcement learning training processes to define what playing at different levels mean, where behaviors can be tuned by using an AI approach.
For example, one component may describe the aggressiveness of an agent in a racing game. The reward function may be based on the number of collisions made by the agent during a race, where a reward may be provided. Thus, a higher reward may be provided for more collisions when an aggressive driving agent is desired. A policy can be trained across a range of rewards so that the user, at run time, can select an aggressiveness component for the agent against which they are racing. Thus, instead of training multiple policies, for example, for low, medium and high aggressiveness, a single policy can be realized for this component, as well as additional components, for the agent.
As discussed in greater detail below, aspects of the present invention provide the ability to train an agent across a plurality of weights for various components of the domain, such as a racing game domain, where the weight becomes an input to the neural network, so that the weight can be chosen at run time by the user. In some embodiments, the weighting can be provided as an input to both the policy and the Q-function. While inputting the weighting into the Q-function is not required in all aspects of the present invention, such an input may help achieve the learning during the training in a more stable fashion.
In some embodiments, the agent is trained in a training step so that the user can select parameters during game play. In other embodiments, the agent may also be trained during game play as well, where the user can pick parameters for game play and these results can be fed back into neural network to update the policy as needed. For example, if the user selects an aggressiveness level of 5 (on a scale of 1 to 10, for example), and the agent performs with a number of collisions greater than that desired for the selected level, this information may be fed back to the network so that the policy may be updated appropriately to further limit the number of collisions at this selected level for this selected component.
Referring to
where wi is a scalar component weight and Ri(s, a) is the reward function for the i-th component. Usually, RL applications keep w fixed as a single scalar vector during an experiment and often search for the w best suited for their application over multiple experiments. Aspects of the present invention, however, can train an agent over a continuous goal space including parameterized reward functions represented by their weights.
Instead of keeping w fixed, aspects of the present invention can sample one or multiple dimensions i of w from a continuous (e.g. uniform) distribution. This subset of non-fixed dimensions of w can be denoted as ŵ. To put more emphasis on a specific weight or segment of a weight range and improve the agent's performance for that segment, skewed distributions may be used, such as a log-uniform distribution. For the basic version of the approach, ŵ can be sampled once per training rollout at the beginning of the episode and it can be kept fixed thereafter.
As a possible extension, it is proposed to additionally repeatedly re-sample ŵ during a rollout, such that the agent becomes robust to reward function changes during ongoing trajectories.
To inform the trained agent of the reward function it is operating under, ŵ can be provided as an additional input to both the policy (actor) and value functions (critic) of the training algorithm. The neural network policy can be updated from π(s) to π(s,ŵ) and the action-value function Q(s, a) to Q(s, a, ŵ) by concatenating the non-fixed reward component weights with the rest of the inputs.
When evaluating the agent's policy at inference time, ŵ can then be set to any of the weights covered during training and the agent can adapt its behavior accordingly and strive to behave optimally under the represented reward function without any retraining.
Aspects of the present invention provide a set of reward parts that can be used in combination with the previously defined reward weight-based goal formulation for the domain of designing autonomous opponents for racing games. Those reward parts allow the encoding of different desired behavior types into the reward function. The progress along the racing course centerline achieved by the agent can be used in an environment step as a base reward to motivate the agent to finish the race course as quickly as possible. The sampled weights for the novel reward parts ŵ then describe the importance of those parts in relation to the fixed weight progress reward. The following reward parts are proposed to be used in addition to the progress reward, either individually or in combination with each other: (1) A penalty for the degradation of the agent's car tires during an environment step. This motivates the agent to save its tires and drive more conservative trajectories similar to careful human drivers, resulting in slower trajectories; (2) A penalty for the agent's fuel use during an environment step. This motivates the agent to drive more economical and smoother trajectories with less extreme changes in speed; (3) A linear penalty for tire slip ratios and angles. This motivates the agent to reduce the chance of slipping and therefore drive safer trajectories similar to those of novice or careful human drivers, that brake earlier before heading into a curve and accelerate later when exiting the curve, which results in a slower trajectory overall; (4) A linear positive reward for tire slip ratio and angle. This motivates the agent to drift, especially in curves where drifting still allows achieving a relatively high progress reward; (5) An edge distance penalty that linearly increases with the agent's proximity to the racing track edge. By setting the weight of this reward, one can influence how much of the track's width the agent uses; (6) A set of reward parts penalizing an agent for driving in corresponding slices of the track defined by the distance to the centerline, e.g., centerline to 2 m from centerline, 2 m to 4 m, 4 m to 6 m, and the like. Based on the weights configured for those reward parts during inference, a game designer can influence the agents driving line post training; (7) A passing reward with independently weighted positive and negative parts for overtaking and being overtaken by other cars respectively. By varying the weights for these rewards individually, an agent can be trained either keen on passing opponents or defending against being overtaken; (8) A penalty on the change in steering angle during an environment step. This motivates the agent to reduce steering changes, resulting in smoother steering movements and earlier braking points to manage driving through curves with smaller steering ranges; (9) A penalty for colliding with other vehicles. Based on this penalty's weight we can tune the agent's aggressiveness and assertiveness nearby other drivers; and (10) A penalty for the car driving off course, either measured by the center of the car or its tires. Based on this penalty's weight, one can tune how often an agent violates track boundaries or, when used with a relatively high weight, how close to the track's edge an agent drives.
While the above disclosure focuses on the domain of a racing game, it should be understood that aspects of the present invention may be applied to AI agents used in various different domains. For example, the AI agent may be one in an animation or locomotion domain, where, for example, one of the components may be provided to change the weight of an energy cost to create slower, faster or more expressive walking of the AI agent. As another non-limiting example, in a fighting game, an AI agent may be trained with various proficiencies on various weapons, where a single policy may be able to provide a range of proficiencies on weapons, such as a bow, sword, axe or the like.
The computer platform 300 may include a central processing unit (CPU) 302, a hard disk drive (HDD) 304, random access memory (RAM) and/or read only memory (ROM) 306, a keyboard 308, a mouse 310, a display 312, and a communication interface 314, which are connected to a system bus 316.
In one embodiment, the HDD 304, has capabilities that include storing a program that can execute various processes, such as the AI agent training engine 350, in a manner to perform the methods described herein.
All the features disclosed in this specification, including any accompanying abstract and drawings, may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.
Claim elements and steps herein may have been numbered and/or lettered solely as an aid in readability and understanding. Any such numbering and lettering in itself is not intended to and should not be taken to indicate the ordering of elements and/or steps in the claims.
Many alterations and modifications may be made by those having ordinary skill in the art without departing from the spirit and scope of the invention. Therefore, it must be understood that the illustrated embodiments have been set forth only for the purposes of examples and that they should not be taken as limiting the invention as defined by the following claims. For example, notwithstanding the fact that the elements of a claim are set forth below in a certain combination, it must be expressly understood that the invention includes other combinations of fewer, more or different ones of the disclosed elements.
The words used in this specification to describe the invention and its various embodiments are to be understood not only in the sense of their commonly defined meanings, but to include by special definition in this specification the generic structure, material or acts of which they represent a single species.
The definitions of the words or elements of the following claims are, therefore, defined in this specification to not only include the combination of elements which are literally set forth. In this sense it is therefore contemplated that an equivalent substitution of two or more elements may be made for any one of the elements in the claims below or that a single element may be substituted for two or more elements in a claim. Although elements may be described above as acting in certain combinations and even initially claimed as such, it is to be expressly understood that one or more elements from a claimed combination can in some cases be excised from the combination and that the claimed combination may be directed to a subcombination or variation of a subcombination.
Insubstantial changes from the claimed subject matter as viewed by a person with ordinary skill in the art, now known or later devised, are expressly contemplated as being equivalently within the scope of the claims. Therefore, obvious substitutions now or later known to one with ordinary skill in the art are defined to be within the scope of the defined elements.
The claims are thus to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what can be obviously substituted and also what incorporates the essential idea of the invention.
