RISK-SENSITIVE INVESTMENT STRATEGIES UNDER PARTIALLY OBSERVABLE MARKET CONDITIONS

Abstract

System, method and computer program product for modelling Risk-Sensitive Partially-Observable Markov Decision Processes (POMDPs), e.g., in a high-risk domain such as financial planning and solving such equations exactly, such that agents maximize the expected utility of their actions. The system and method employs an exact algorithm for solving Risk-Sensitive POMDPs, for piecewise linear utility functions, by representing underlying value functions with sets of piecewise bilinear functions—computed using functional value iteration—and pruning the dominated bilinear functions using efficient linear programming approximations of underlying non-convex bilinear programs. Considering piecewise linear approximations of utility functions, (i) there is defined the Risk-Sensitive POMDP model that incorporates value functions V(b,w) where argument “b” is a belief state and argument “w” is a continuous wealth dimension; (ii) derive the fundamental properties of the underlying value functions and provide a functional value iteration technique to compute them; and (iii) determine the dominated value functions, to speed up the algorithm.

Description

FIELD OF INVENTION

The present invention relates generally to financial planning and investing, and particularly, to a system and method for devising investment strategies and determining an optimal investment strategy in accordance with an expected risk sensitivity at a particular point in time.

BACKGROUND

Recent years have seen an unprecedented rise of interest in decision support systems that help investors to choose an investment strategy to maximize their returns. In particular, Partially Observable Markov Decision Processes (POMDPs) (see, e.g., E. J. Sondik, entitled The Optimal Control of Partially Observable Markov Processes, Ph.D Thesis, Stanford University, 1971) have received a lot of attention due to their ability to provide multistage strategies that address the uncertainty of the investment outcomes and the uncertainty of market conditions head-on.

Yet, POMDP solvers (see, e.g., M. Hauskrecht, entitled Value-function approximations for POMDPs, JAIR, 13:33-94, 2000; Z. Feng and S. Zilberstein entitled Region-based incremental pruning for POMDPs in UAI, pages 146-15, 200; and, J. Pineau, G. Gordon, and S. Thrun entitled PBVI: An anytime algorithm for POMDPs, IJCAI, pages 335-344, 2003) typically maximize the expected utility of the investments. In contrast, in high-stake domains such as financial planning, it is often imperative to find an optimal investment strategy that maximizes the expected “utility” of the investments, for non-linear utility functions that characterize the investor attitude towards risk. While there has been demonstrated how to solve multistage stochastic optimization problems where risk-sensitivity is expressed via utility functions, this was only for problems characterized by fully observable market conditions.

It would be highly desirable to provide a system and method that enables the generation of a theoretic model for risk-sensitive financial planning under partially observable market conditions and the solution of such model that accounts for risk sensitivity.

Currently, there are no algorithms known in the art that can provide an optimal POMDP solution that accounts for risk sensitivity.

It would be highly desirable to provide a system and method that enables the generation of a theoretic model for risk-sensitive financial planning under partially observable market conditions and the solution of such model that accounts for risk sensitivity.

SUMMARY

The present invention addresses the above-mentioned shortcomings of the prior art approaches by first defining Risk-Sensitive POMDPs, and generating a novel decision theoretic model for risk-sensitive financial planning under partially observable market conditions.

In one aspect, by considering piecewise linear approximations of utility functions, the method implements a functional value iteration method using a “solver” to solve Risk-Sensitive POMDPs optimally by computing the underlying value functions exactly, through the exploitation of their piecewise bilinear properties. In one aspect, the value functions are derived analytically using a Functional Value Iteration algorithm.

Further to this aspect, to speed up the implemented Risk-Sensitive POMDPs solver, the system and method performs finding and pruning the dominated investment strategies using efficient linear programming approximations to the underlying non-convex bilinear programs. That is, by deriving the fundamental properties of the underlying value functions, the method provides a functional value iteration technique to compute them exactly, and further, provides an efficient procedure to determine the dominated value functions, to speed up the algorithm.

In one aspect, there is provided a system, method and computer program product for determining an investment strategy for a risk-sensitive user. The method comprises: modeling an user's attitude towards risk as one or more utility functions, the utility functions, the utility function transforming a wealth of the user into a utility value; generating a risk-sensitive Partially Observable-Markov Decision Process (PO-MDP) based on the one or more utility functions; and, implementing Functional Value Iteration for solving the risk sensitive PO-MDP, the solution determining an action or policy calculated to maximize an expected total utility of an agent's actions at a particular point in time acting in a partially observable environment.

Further to this aspect, the generating of the risk-sensitive PO-MDP comprises: generating an expected utility function V_Uⁿ(b,w) for 0≦n≦N,b∈B,w∈Wⁿ where Wⁿ denotes the set of all possible user wealth levels in decision epoch n; and, maximizing the expected utility function V_Uⁿ(b,w) for a user when commencing action a∈A, where A is a set of Actions, in decision period n in a belief state b with a wealth level w.

In a further aspect, there is provided a system for determining an investment strategy for a risk-sensitive user comprising: a memory; a processor in communications with the memory, wherein the system performs a method comprising: modeling an user's attitude towards risk as one or more utility functions, the utility functions the utility function transforming a wealth of the user into a utility value; generating a risk-sensitive Partially Observable-Markov Decision Process (PO-MDP) based on the one or more utility functions; and, implementing Functional Value Iteration for solving the risk sensitive PO-MDP, the solution determining an action or policy calculated to maximize an expected total utility of an agent's actions at a particular point in time acting in a partially observable environment.

A computer program product is provided for performing operations. The computer program product includes a storage medium readable by a processing circuit and storing instructions run by the processing circuit for running a method. The method is the same as listed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will become apparent to one skilled in the art, in view of the following detailed description taken in combination with the attached drawings, in which:

FIG. 1 an example problem set-up for planning under uncertainty, e.g., in a financial planning domain, by incorporating risk sensitive planning in partially observable domains;

FIGS. 2A-2C depict a methodology 100 employed for devising an optimal single or multi-stage investment strategy in one example;

FIG. 3 depicts example utility functions that may be constructed to represent a particular entity's attitude toward risk in an example embodiment;

FIG. 4A depicts in an example implementation results 350 showing a plot of epsilon ε (plotted on the x-axis) vs. runtime (e.g., in seconds on a logarithmic scale), and vs. the solution quality (plotted on the y-axes) in example results 360 shown in FIG. 4B;

FIG. 5 depicts conceptually use of functional value iteration technique 375 for solving Risk-Sensitive POMDPs to provide action(s) designed to achieve a maximized expected utility at an example chosen decision epoch;

FIG. 6 is a visual representation of the set-up problem (S, A, P, O, R, Z, U) of the risk sensitive PO_MDP model 200;

FIG. 7 graphically depicts example solver results 220 that can be used for extracting an agent policy, e.g., an investment action to perform;

FIG. 8 graphically depicts solver results for example strategies (e.g., two different actions) as two example value functions 275a, 275b to maximize an expected utility based on proposed strategy.

FIG. 9 illustrates an exemplary hardware configuration for implementing the flow charts depicted in FIGS. 2A- 2C in one embodiment.

DETAILED DESCRIPTION

In one aspect, there is provided a system, method and computer program product that provides and solves for a Risk-sensitive investor, an optimal investment strategy. In one embodiment, the system and method allows for Multistage investment strategies. The system and method operates to estimate market state from noisy observations, and, handles partially observable market states. Thus, in one aspect, to estimate the market state from noisy observations, the method of the invention employs modeling the data as a Partially Observable Markov Decision Process (PO-MDP).

FIG. 1 provides an illustrative example of a problem 10 set-up for planning under uncertainty in partially observable domains, for instance, in a financial planning domain, by incorporating risk sensitive planning in partially observable domains. In the example, a decision is to be made as to whether to invest the current wealth 15, e.g., $1000. This decision has to be made considering that the state of the market 17 is uncertain, e.g., depicted as a probability of being in either of two market states 19, e.g., and comprising a value 20% good, and 80% bad, and the return on investment is also uncertain. FIG. 1 provides further details on this example setting and, for purposes of explanation, focus is made on a single decision. However, the invention is applicable to general problems where a sequential set of decisions need to be made.

In one embodiment, there are two ways to make decisions in such settings: (a) Expected value maximization 20 which is the risk neutral way to make decisions, i.e., by not considering that people have various attitudes towards risk. Thus, there is always the same decision made by this method. As shown in FIG. 1, maximizing expected value provides a decision to not invest 25; (b) Expected utility maximization 40 which mechanism is sensitive to the risk attitude of the person and as shown, depending on whether the person is risk seeking 33 (as indicated from utility function 43 ), or risk averse 35 (as indicated from utility function 45 ) which depicts a slower rate of utility for the same wealth, the decision appropriately changes. For example, the options may result in a decision to invest (e.g., in a good market) or may result in a decision to not invest (e.g., in a bad market). The invention answers given the expected wealth and stated market conditions which action or policy to pursue (e.g., given a bad market or good market in the example shown in FIG. 1).

As utility theory defines utility functions as transforming the current wealth of an agent (its initial wealth plus the sum of the immediate rewards it received so far) into a utility value, the shape of the utility function can be used to define the agent attitude towards risk. To compute optimal policies for such risk-sensitive agents, acting in partially observable environments, the finite horizon POMDPs may be solved that maximize the expected total utility of agent actions. On account of being sensitive to risk attitudes, these planning problems are referred to as Risk-Sensitive POMDPs characterized as comprising the following: S is a finite set of discrete states of the process; A is a finite set of agent actions. The process starts in some State s₀∈S and runs for N consecutive decision epochs. In particular, if the process is in state s∈S in decision epoch 0≦n≦N, the agent controlling it chooses an action a∈A to be executed next. The agent then receives the immediate reward R(s,a) while the process transitions with probability P(s′|s,a) to state s′∈S at decision epoch n+1. Otherwise, in decision epoch n=N, the process terminates.

The utility of the actions that the agent has executed is then a scalar

U(w₀+Σ_n=0^N−1r_n)

where w₀ is the initial wealth of the agent, U is the agent utility function and r_n is the immediate reward that the agent received in decision epoch n. The goal of the agent is to devise a policy π that maximizes its total expected utility:

E[U(w₀+Σ_n=0^N−1r_n)|π].

What further complicates the agent's search for policy “π” is that the process is only partially observable to the agent. That is, the agent receives noisy information about the current state s∈S of the process and can therefore only maintain the current probability distribution b(s) over states s∈S (referred to as the agent belief state). When the agent executes some action a∈A and the process transitions to state s′, the agent receives with probability O(z|a, s′) an observation z from a finite set of observations Z. The agent then uses z to update its current belief state b, as will be described in greater detail herein below. In the following, B denotes an infinite set of all possible agent belief states and b₀∈B is the agents' starting belief state (e.g., unknown at the planning phase).

Additionally, W:=∪_0≦n≦NWⁿ is the set of all possible agent wealth levels where Wⁿ denotes the set of all possible agent wealth levels in decision epoch n. For the initial range of agent wealth levels W⁰:=[w⁰, w⁰] there is determined Wⁿ=[wⁿ, wⁿ] where wⁿ=wⁿ⁻¹+min_s∈S,a∈AR(s,a) and wⁿ= wⁿ⁻¹+max_s∈S,a∈AR(s,a), for n=1, . . . ,N. It is noted that W⁰⊂W¹⊂ . . . ⊂W^N. A policy π of the agent therefore indicates which action π(n,b,w)∈A the agent should execute in decision epoch n, belief state b, with wealth level w, for all 0≦n≦N , b∈B , w∈Wⁿ.

FIGS. 2A-2C provide a methodology 100 for devising an optimal single or multi-stage investment strategy. The method may be run in a computer or like processing device and a suitable storage media, e.g., a computer program product, may include instructions configured for devising an optimal single or multi-stage investment strategy.

In the method 100 for providing or devising an optimal single or multi-stage investment strategy, at 102, an entity, a user, a business organization, a business target, an agent, to construct one or more utility functions. These utility functions are of a shape that can represent the user, e.g., agent's, attitude towards risk and the PO-MDPs solver framework is used to maximize the expected total utility (as opposed to expected total reward) of agent actions. For purposes of illustration, FIG. 3 shows several example utility functions labeled 50A-50E constructed by an entity, e.g., a user, a business organization, etc., that depicts a particular user's or business unit's attitudes toward risk with each function depicted as a plot of perceived expected value or figure of merit (e.g. (utility) vs. potential wealth accumulation. For example, utility function 54, a continuous function, U(w), is a plot depicting an example situation where the company sets a target to accumulate wealth of −10 or better (more), as there is perceived no extra utility in getting more money. However, in example utility function 58 that a company may construct, there may be three (3) targets (stages) indicated: e.g., to obtain a target wealth of, e.g., −17 or more, obtain a target wealth −10 or more, or −3 or more.

In the set-up of the PO_MDP, the elicited utility function(s) U(w) that express the investor's attitude towards risk by mapping all attainable wealth levels w to their utility, as perceived by a user, e.g., an investor, an agent, for example, are input to a computer or like processing device such as described with respect to FIG. 9 for processing thereof.

As shown in FIG. 2A, at 105, for a given financial domain, the method formulates a Risk-Sensitive PO_MDP problem.

Then, this Risk-Sensitive POMDP is solved. That is, there is determined what action (policy) a∈A should the investor execute in decision epoch n∈[0, 1, . . . , N], with wealth level w∈[w_min, w_max], if the investor believes that the probability that the market is in state s is b(s), for all s∈S. As shown in FIG. 2A, in another aspect, the solver implemented in generating the solution to the PO_MDP may be accelerated (speed-up) at 170 by pruning dominated strategies as will be described in greater detail hereinbelow.

The processing at 110, FIG. 2A is now described in view of FIG. 2B processing where, in order to perform step 110, there is performed: at step 120, the generation of the expected utility function V_Uⁿ(b,w) to be maximized for the investor if investor starts acting in decision epoch n in belief state b (distribution over states s∈S) with wealth level w. Then, at 125, the V_Uⁿ(b,w) function is maximized by executing an action π*(n,b,w) that is computed in accordance with equation 1) as follows:

$\begin{array}{cc}\arg \underset{a\in A}{\max }\left\{\sum _{z\in Z}P\left(zb,a\right)V_U^{n+1}\left(T\left(b,a,z\right),w+R\left(b,a\right)\right)\right\}& 1)\end{array}$

where P(z|b,a)=Σ_s′∈SO(z|a, s′)Σ_s∈SP(s′|s,a)b(s) is the probability of observing z after executing action a from belief state b, R(b,a):=Σ_s∈Sb(s)R(s,a) is the expected immediate reward that the agent will receive for executing action a in belief state b and T(b,a,z) is the new belief state of the agent after executing action a from belief state b and observed z. Formally, for each s′∈S it holds that:

T(b,a,z)(s′)=[O(z|a,s′)/P(z|b,a)]Σ_s∈SP(s′s,a)b(s).

Hence, to find the optimal policy, π*, value iteration is employed to calculate values V_Uⁿ(b,w) for all 0≦n≦N, b∈B,w∈Wⁿ. Value iteration calculates these values for n=N,N−1, . . . ,0. Specifically, as follows from step 150, FIG. 2C, for n=N the process terminates and thus:

V _U ^N(b,w)=U(w)   2)

for all w∈Wⁿ, b∈B. Otherwise, for all 0≦n≦N,

$\begin{array}{cc}{V}_U^n\left(b,w\right)==\underset{a\in A}{\max }\left\{\sum _{z\in Z}P\left(zb,a\right)V_U^{n+1}\left(T\left(b,a,z\right),w+R\left(b,a\right)\right)\right\}& 3)\end{array}$

for all b∈B and w∈Wⁿ. In the following, values of V_Uⁿ(b,w) are grouped over all (b,w)∈B×W into value functions V_Uⁿ:B×W→, for each 0≦n≦N. Note, that computing value functions V_Uⁿ from value functions V_Uⁿ⁺¹ exactly is difficult because B and W are infinite. In addition, POMDP solution techniques that already handle an infinite B—are not applicable for solving Risk-Sensitive POMDPs as they do not handle an infinite W.

The functional value iteration technique for solving Risk-Sensitive POMDPs exactly is now described according to one embodiment. This technique backs up utility functions (unlike just reward values in value iteration) defined on the wealth over the entire time horizon. The method iteratively constructs the finite partitioning of the B×W search space into regions where the value functions can be represented with point based policies, a point based policy being a mapping from the observations received so far to an action that should be executed next. For example, as shown in FIG. 5, using functional value iteration technique 375 for solving Risk-Sensitive POMDPs, there is depicted conceptually an example point based policy 380 resulting from performing actions and observing over 3 decision epochs (n=2) two possible observations z1, z2. In the example depiction, the point based policy 380a determines for the third epoch n=2 a policy of actions A1, A2 dependent upon the observations (z1, z2) resulting from performing action A1 in decision epoch n=1, or a point based policy 380b determined for the third epoch n=2 an action A2 dependent upon the observations (z1, z2) resulting from performing action A2 in prior decision epoch n=1.

In one embodiment, if there is only two states, then a belief state b belongs to a set [0,1] =B; a wealth interval on the other hand is [W_min,W,_max]=W. Thus, a “whole” region is B×W can be partitioned in multiple ways, e.g., into four sub-regions:

[0,0.5]×[W_min, (W_min+W_max)/2]

[0,0.5]×[W_min+W_max)/2, W_max]

[0.5,1]×[W_min, (W_min+W_max)/2]

[0.5,1]×[W_min+W_{max )/}2, W_{max ]}

To this end, Zⁿ is denoted as a set of agent observation histories of length less than “n”. Also, for each decision epoch 0≦n≦N, there is defined a point based policy {dot over (π)}ⁿ as a function

{dot over (π)}ⁿ:Z^N−n→A   4)

and the expected utility to go of {dot over (π)}ⁿ at some belief state and wealth level pair (b,w)∈B×Wⁿ as a value (i.e., a function over B×Wⁿ) set forth according to equation 5 ) as follows:

$\begin{array}{cc}\upsilon 〈{\stackrel{.}{\pi }}^n〉\left(b,w\right):=E\left[U\left(w+\sum _{n^{\prime }=n}^{N-1}r_{n^{\prime }}\right){\stackrel{.}{\pi }}^n,b_0=b\right]).& 5)\end{array}$

Letting {{dot over (π)}_iⁿ}_i∈I(n) be a collection of point-based policies such defined, for a decision epoch n, then any policy π can be represented as some (possibly infinite) collection of point-based policies. For example, to represent n in decision epoch n, a different point-based policy {dot over (π)}_iⁿ may be maintained for each (b,w)∈B×Wⁿ. In particular, to represent π* in decision epoch n, there may be maintained a different point-based policy argmax_{{dot over (π)}in}υ{dot over (π)}_iⁿ(b,w) for each (b,w)∈B×Wⁿ. A finite collection {{dot over (π)}_iⁿ}_i∈I(n) is sufficient to represent π*, for each 0≦n≦N. That is, there exists a finite partitioning {Y_iⁿ}_i∈I(n) of B×Wⁿ and a finite collections {{dot over (π)}_iⁿ}_i∈I(n) such that υ{dot over (π)}_iⁿ(b,w)=V_Uⁿ(b,w) for all (b,w)∈Y_iⁿ.

In one aspect of the invention, finite collections {{dot over (π)}_iⁿ}_i∈I(n) for 0≦n≦N that represent π* are computed. The technique of the invention approximates that the utility function U(w) is piecewise linear over w∈W^N (or, that it has already been approximated with a piecewise linear function with a desired accuracy). Specifically, given that there exists wealth levels w^N=w₁<. . . <w_K= w^N and pairs of constants (C₁, D₁), . . . (C_K,D_K) such that U(w)=C_kw+D_k for all w∈[w_k,w_k+1) over all 1≦k≦K.

According to the invention, for such U, as is proven by induction analysis, the following holds for all 0≦n≦N:

1. The value function V_Uⁿ is represented by a finite set of functions {υ{dot over (π)}_iⁿ}_i∈I(n). That is, there exists a partitioning {Y_iⁿ}_i∈I(n) of B×Wⁿ and a set of point-based policies {{dot over (π)}_iⁿ}_i∈I(n) such that for all (b,w)∈B×Wⁿ there exists i∈I(n) such that (b,w)∈Y_iⁿ and V_Uⁿ(b,w)=υ{dot over (π)}_iⁿ(b,w)=max_i′∈I(n)υ{dot over (π)}_iⁿ(b,w).2. For all i∈I(n), υ{dot over (π)}_iⁿ is piecewise bilinear. That is, there exists a finite partitioning {B×W_i,kⁿ}_k∈I(n,i) of B×Wⁿ such that W_i,kⁿ is a convex set and for all (b,w)∈B×W_i,kⁿ, υ{dot over (π)}_i^n(b,w)=Σ_s∈Sb(s)(c_i,k,sⁿw+d_i,k,sⁿ), for all k∈I(n,i);3. For all i∈I(n), υ{dot over (π)}_iⁿ can be derived from the set of functions {υ{dot over (π)}_i′ⁿ⁺¹}_{i′∈I(n+1)}.

Induction Analysis

As part of reduction analysis, induction holds for n+1 and it also holds for n. To this end, from Equation (3), as V_Uⁿ(b,w) is calculated by:

$\underset{a\in A}{\max }\left\{\sum _{z\in Z}P\left(zb,a\right)V_U^{n+1}\left(T\left(b,a,z\right),w+R\left(b,a\right)\right)\right\}$

which calculation is broken into five stages:

First, as shown in the Appendix, there is calculated, in a first stage,

• • V_U,a,zⁿ(b,w):=V_Uⁿ⁺¹(T(b,a,z),w) where V_Uⁿ⁺¹ is represented by {υ{dot over (π)}_iⁿ⁺¹}_i∈I(n+1) from the induction assumption. Then, in a second stage, there is derived
  • V_U,a,zⁿ(b,w):=P(z|b,a)V_U,a,z(b,w) and then, in a third stage,
  • V_U,aⁿ(b,w):=Σ_z∈ZV_U,a,zⁿ(b,w). Then, at a fourth stage, there is derived
  • V_U,aⁿ(b,w):=V_U,aⁿ(b,w+R(b,a)). The proof of the induction step is concluded at a fifth stage by calculating V_Uⁿ(b,w):=max_a∈AV_Uⁿ(b,w) where V_Uⁿ is represented by {υ{dot over (π)}_iⁿ}_i∈I(n).

Thus, as shown in FIG. 7, the Functional Value Iteration technique for solving the Risk-sensitive PO-MDP exactly, results in a solution set of value functions for each decision stage “n” (the solution is defined for all decision epochs n=1 , . . . ,N). By considering piecewise linear approximations of utility functions (FIG. 3), the functional value iteration method solves Risk-Sensitive POMDPs optimally by computing the underlying solution set of value functions exactly, through the exploitation of their piecewise bilinear properties.

Referring to FIG. 2C, there is depicted a methodology 150 for solving the underlying value functions exactly through the exploitation of their piecewise bilinear properties. As shown at step 150, there is depicted a first step of setting V_U^N(b,w) equal to the maximum expected utility U(w) for the investor if its starts acting in decision epoch n in belief state b (distribution over states s∈S) with wealth level w. The process enters an iterative loop at step 155, for example, a “for” loop setting iterations for each decision epoch s n=N−1 to n=0, for example. At each decision epoch n=N−1 to n=0 the following is performed: 1) at 160, FIG. 2A, representing V_Uⁿ⁺¹(b,w) using a set of bilinear functions γⁿ⁺¹:={{dot over (π)}_iⁿ⁺¹(b,w) }_i∈I(n+1). Then, at 165, the bilinear functions from γⁿ⁺¹ are used to construct the set of bilinear functions γⁿ that jointly represent V_U ⁿ(b,w).

The operation to construct the set of bilinear functions γⁿ is performed by a Linear/Integer program “solver”, such as ILOG CPLEX™ available from International Business Machines, Inc.) embodied by a programmed computing system (e.g., a computing system 400 as shown in FIG. 9). Particularly, the inputs to the solver are:

N=The number of decision epochs;U=The agent utility function(s) that maps the agent wealth w to its utility; U(w) is a piecewise linear approximation of an arbitrary utility function elicited from a user, e.g., an investor and is specified by constants C_k, and D_k, k=1, . . . , K, as explained in greater detail herein below.

As shown in FIG. 6, the set-up problem (S, A, P, O, R, Z, U) of the POMDP model 200 comprises the following:

• • S=the set of states; for example, S={s1,s2} where s1 denotes a “market is bad” state and s2 denotes “market is good” state. There can be more than two states, e.g., if a state describes multiple markets (that can be good/bad);
  • A=the set of actions (e.g. invest/do not invest in company X/Y/Z etc.);
  • P=the state to state transition function;
  • Z=the set of observations;
  • O=the observation function; and,
  • R=the reward function.

An example data structure to represent these solver inputs is therefore a tuple (N,U,S,A,P,Z,O,R) where N is an integer, U is a piecewise linear function on domain (min_wealth, max_wealth), S,A,O are binary vectors to give unique identifiers to states, actions and observations respectively. P:S×A×S →[0,1] is a state to state transition function, O:S×A×Z→[0,1] is an observation function and R:S×A→[reward_min, reward_max] is a reward function.

The equations for processing these inputs by the solver are programmed into the solver and are computed according to the proof by induction provided in the Appendix. Additionally, the solver proceeds by computing the value functions Vⁿ(b,w) starting from n=N, then n=N−1, . . . , and finally n=0. As soon as V⁰(b,w) is found, the agent knows what action to execute in the starting decision epoch.

In solving the equations below, the following are defined:

n is the current epoch;

w is the wealth level;

s denotes some state;

b is a probability distribution over states, i.e., the agent current belief state;

b(s) is a an agent belief that the system is in state s with a certain probability, for all states from the set of states S. As an example, two states, sb and sg are considered such that sb=market is bad, and sg=market is good. Then b=(0.2, 0.8) means that the agent beliefs that the current system state is sg with probability b(sg)=0.2, and that the current system state is sb with probability b(sb)=0.8;

b is a belief variable;

w is a wealth variable;

(b,w) is a feasible solution to the

(b′ , x) is a feasible solution corresponding to (b,w) where (b′:=b,x:=bw);

x=[x(s)],_s∈S is a vector.

Program (17) relaxes Program (16b) because for any there exists a corresponding feasible solution (b′:=b,x:=bw)

c and d (or the variations thereof, with various indices) are constants.

V(b,w) is the value function returned by the solver hat is represented using sets of bilinear functions.

The method includes implementing calculations performed by solver. When the algorithm starts, the known constants are the constants C_k and D_k k=1,2, . . . , K that specify the piecewise linear utility function U (defined in each of the K wealth intervals as a linear function C_k w+D_k). In the description of the method, auxiliary constants c and d are introduced (as set forth in the staged operations 1,2,3,4,5 in the Appendix).

The method includes:

• • 1. Calculating, by the solver, during a stage 1 calculation, the following equation (19) from Lemma 1, Appendix:

$\begin{array}{c}{\upsilon }_{a,z,i}^n\left(b,w\right):=\upsilon 〈{\stackrel{.}{\pi }}_i^{n+1}〉\left(T\left(b,a,z\right),w\right)\\ =\sum _{s\in S}b\left(s\right)\sum _{s^{\prime }\in S}P\left(s^{\prime }s,a\right)O\left(za,s^{\prime }\right)\left(c_{i,k,s^{\prime }}^{n+1}w+d_{i,k,s^{\prime }}^{n+1}\right),\\ =\sum _{s\in S}b\left(s\right)\left(c_{a,z,i}^{n,k,s}w+d_{a,z,i}^{n,k,s}\right)\end{array}$

for constants c_a,z,i^n,k,sΣ_s′∈SP(s′|s,a)O(z|a,s′)c_i,k,s′ⁿ⁺¹ and, d_a,z,i^n,k,sΣ_s′∈SP(s′|s,a)O(z|a,s′)d _i,k,s′ⁿ⁺¹ where these constants c_a,z,i^n,k,s and d_a,z,i^n,k,s are obtained by computer system from utility functions, observed data and belief states. For example, constants cⁿ⁺¹_i,k,s and dⁿ⁺¹_i,k,s are obtained by the computer system from utility functions (when n=N) or, from the previous algorithm iteration (when n<N). This calculation exhibits that function υ_a,z,iⁿ(b,w) from a stage 1 calculation is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹.

• • 2. Calculating, by the solver, during a stage 2 calculation, the following equation (9) from Stage 2, Appendix:

$\begin{array}{c}{\stackrel{\_}{\upsilon }}_{a,z,i}^n\left(b,w\right):=P\left(zb,a\right){\upsilon }_{a,z,i}^n\left(b,w\right)\\ =P\left(zb,a\right)\sum _{s\in S}b\left(s\right)\left(c_{a,z,i}^{n,k,s}w+d_{a,z,i}^{n,k,s}\right),\\ =\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,z,i}^{n,k,s}w+{\stackrel{\_}{d}}_{a,z,i}^{n,k,s}\right)\end{array}$

for all (b,w)∈B×W_iⁿ⁺¹, k∈I(n+1,i) where c_a,z,i^n,k,s=P(z|b,a)c_a,z,i^n,k,s and d_a,z,i^n,k,s=P(z|b,a)d_a,z,i^n,k,s are constants.

• • 3.Calculating, by the solver, after a stage 2 calculation, the following equation (21) from Lemma 2, Appendix:

${\upsilon }_{a,i,k}^n\left(b,w\right):=\sum _{s\in S}b\left(s\right)\left(c_{a,i}^{n,k,s}w+d_{a,i}^{n,k,s}\right),$

for all (b,w)∈B×Wⁿ⁺¹, where constants c_a,i^n,k,s:=Σ_z∈Zc_a,z,i(z)^n,k(z),s, and d_a,i^n,k,s:=Σ_z∈Zd_a,z,i(z)^n,k(z),s. 4. Calculating, by the solver after a stage 3 calculation, the following equation (24 ) from Lemma 3, Appendix:

${\stackrel{\_}{\upsilon }}_{a,i,k}^n\left(b,w\right):=\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,i}^{n,k\left(s\right),s}w+{\stackrel{\_}{d}}_{a,i}^{n,k\left(s\right),s}\right),$

for all (b,w)∈B×Wⁿ, where c_a,i^n,k(s),s:=c_a,i^n,k(s),s and d_a,i^n,k(s),s:=d_a,i^n,k,(s),sR(s,a) are constants.5. Then, calculating, by the solver, the following equation (25 ) from Lemma 3, Appendix:

$\begin{array}{c}{\stackrel{\_}{\upsilon }}_{a,i}^n\left(b,w\right):={\upsilon }_{a,i}^n\left(b,w+R\left(b,a\right)\right)\\ =\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,i}^{n,k\left(s\right),s}w+{\stackrel{\_}{d}}_{a,i}^{n,k\left(s\right),s}\right)\\ ={\stackrel{\_}{\upsilon }}_{a,i,k}^n\left(b,w\right)\end{array}$

6. Finally, there is calculated by the solver, a calculation of the following equation (15) from Stage 5, Appendix:

$\begin{array}{c}{V}_U^n\left(b,w\right):=\underset{\left(a,i\right)\in I\left(n\right)}{\max }{\stackrel{\_}{\upsilon }}_{a,i}^n\left(b,w\right)\\ \equiv \upsilon 〈{\stackrel{.}{\pi }}_{\left(a,i\right)}^n〉\left(b,w\right)\end{array}$

Therefore, V_U^N(b,w) is represented by a finite set of piecewise bilinear functions Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n)={ υ_a,iⁿ}_(a,i)∈I(n) derived (through stages 1,2,3,4,5, Appendix) from functions {υ{dot over (π)}_i′ⁿ⁺¹}_{i′∈I(n+1)} which proves the claims of the induction step and the whole proof by induction.

Thus, in the method implemented by the solver, the output produced at each of the equations below is a new (temporary) set of bilinear functions, represented using the corresponding new (temporary) constants c and d (with different indices). At the last step, the solver returns the value function V(b,w) at an epoch n that is represented using sets of bilinear functions Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n)={ υ_a,iⁿ}_(a,i)∈I(n), each function represented using calculated constants cⁿ_i,k,s and dⁿ_i,k,s for s from S and k from I(n,i) (i.e., an index from a set I(n,i) of indices associated with decision epoch n and point based policy number i. By examining these value functions V(b,w) the agent can then choose an action that (given b and w) is guaranteed to yield the highest expected total reward (as explained earlier) in decision epoch n.

Thus, when the algorithm terminates, each bilinear function “f_i” from set Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n)={ υ_a,iⁿ}_(a,i)∈I(n) is represented using constants cⁿ_i,k,s and dⁿ_i,k,s for s from S and k from I(n,i)={set of indices}. That is, each function

f _{i=sum {} s}(b(s)*(cⁿ_i,k,s*w+dⁿ_i,k,s))

is bilinear.

FIG. 6 graphically depicts, in an example embodiment, the solver results 220 for extracting an agent policy, e.g., an investment action to perform. That is, to find what action an agent should execute in decision epoch n, with wealth w and belief state b (if it believes that current state is “s” with probability b(s), for all s from S), the agent then looks at the value function Vⁿ(b,w). When the solver terminates, as shown in FIG. 6, each value function Vⁿ(b,w) is represented by a set Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n)={ υ_a,iⁿ}_(a,i)∈I(n) of bilinear functions 250, and each of these bilinear functions has associated with it the first action “a” that should be executed to yield a corresponding bilinear function given a risk of being within a risk sensitive state, e.g., perceived probability between state s 211 and s1212. An agent compares the values of all these bilinear functions at argument (b,w) and may choose to execute action “a” that is associated with the dominant bilinear function at argument (b,w). As an example, action “a” could be: invest/do not invest in X/Y/Z etc. in decision epoch n.

That is, in view of FIG. 6, at an example decision epoch n, a point based policy is given for any pair (b,w). The depth of such policy is the number of decision epochs to go. For example, if N=4 decision epochs, then at decision epoch n=2, a point based policy will ascribe actions to decision epochs 3 and 4. When the user occupies pair (b,w) at decision epoch n, it looks at which bilinear function 250 is dominant for this pair (b,w) at decision epoch n and then retrieves point based policy “π” assigned to this dominant bilinear function (each bilinear function has a point-based policy assigned to it). The first action on the retrieved point-based policy is the action that the agent should perform next. Conversely, if this (retrieved) point-based policy were to be executed many times, it would on average yield utility given by the dominant utility function for pair (b,w).

FIG. 8 graphically depicts solver results for example strategies (e.g., two different actions) as two example value functions 275a, 275b to maximize an expected utility based on proposed strategy. More particularly, the two value functions 275A, 275B depicted in FIG. 8 is associated with a point-based policy. To determine which point based policy an agent would follow when in a pair (b,w), it is determined which utility function is dominant at the pair (b,w).

In a further embodiment, in order to speed up the implemented Risk-Sensitive POMDP solver, the system and method includes finding and pruning the dominated investment strategies using efficient linear programming approximations to underlying non-convex bilinear programs. Thus, referring to FIG. 2C, continuing to step 170, there is performed pruning bilinear functions that are completely dominated by other bilinear functions. The determination as to whether a function υ_a,iⁿ is dominated by another, is now explained:

In one exemplary embodiment, as mentioned in the stages 1,3,5 of the induction proof incorporated herein such as described in Appendix, the solver implements functionality for speeding up the algorithm by pruning, from a set of piecewise bilinear functions, these functions that are jointly dominated by other functions. The solver implemented quickly and accurately identifies if a function is dominated or not. Formally, for a set of piecewise bilinear functions V={υ_i:B×W→R}_i∈I there is determined if some υ_j∈V is dominated, i.e., if for all (b,w)∈B×W there exists υ_i∈V,i≠j such that υ_i(b,w)>υ_j(b,w).

Letting υ_i∈V be piecewise bilinear over B×W , i.e., there is a partitioning {B×W_i,k}_1≦k≦K(i) of B×W such that set W_i,k is convex and υ_i(b,w)=Σ_s∈Sc_i,k^sw+d_i,k^s for all (b,w)∈B×W_i,k, 1≦k≦K(i). Thus, there exists wealth levels w=w_i,0<. . . <w_i,k<. . . <w_i,K(i)= w such that W_i,k=[w_i,k−1,w_i,k] for all 1≦≦k≦K(i) where K(i) is the number of intervals in which the whole wealth interval (W_min, W_max) is split. In determining whether υ_j∈V is dominated functions of V are first split into functions defined over common wealth intervals. Precisely, let W={w_k}_0≦k≦K:=∪_i∈I{w_i,k}_1≦k≦K(i) be a set of common wealth levels where w=w₀<. . . <w_k<. . . w_K= w. For all (b,w)∈B×[w_k−1,w_k], 1≦k≦K then υ_i,(b,w) is represented with υ_i,k(b,w):=Σ_s∈Sc_i,k^sw+ d_i,k^s where c_i,k^s:= c_i,k′^s, d_i,k^s:= d_i,k′^s for k′ such that w∈[w_i,k′−1,w_i,k′], for all i∈I.

υ_j∈V is then not dominated if there exists 1≦k≦K and (b,w)∈B×[w_k−1,w_k] such that for all υ_i∈V, i≠j it holds that υ_i,k(b,w)<υ_j,k(b,w). That is, if for some 1≦k≦K there exists a feasible solution (b,w) to Program

$\begin{array}{cc}\max 0\begin{array}{cc}{\upsilon }_{j,k}\left(b,w\right)-{\upsilon }_{i,k}\left(b,w\right)>0& \forall {\upsilon }_i\in V\\ w_{k-1}\le w\le w_k& \\ \sum _{s\in S}b\left(s\right)=1& \end{array}& 16a)\end{array}$

also written as

$\begin{array}{cc}\max 0\begin{array}{cc}\sum _{s\in S}b\left(s\right)\left(c_{i,j,k}^sw+d_{i,j,k}^s\right)>0& \forall v_i\in V\\ w_{k-1}\le w\le w_k& \\ \sum _{s\in S}b\left(s\right)=1& \end{array}& 16b)\end{array}$

where the program “max O”[+terms]” represents the attempt to maximize the objective function “O”, i.e., an empty/blank objective function; variable b=[b(s)]_s∈S is a vector; c_i,j,k^s:= c_j,k^s− c_i,k^s and d_i,j,k^s:= d_j,k^s− d_i,k^s.

In one embodiment, due to presence of non-linear, non-convex constraints in solving Program (16b), i.e., because of term Σ_s∈Sb(s)c_i,j,k^sw+d_i,j,k^s)>0, υ_i∈V, a solution is to relax the constraints.

However, by relaxing the constraints of Program (16b), the chance of finding a feasible solution (b,w) is increased, thus decreasing the chance of pruning υ_j from V. Therefore such a relaxation may result in keeping in V some of the dominated functions, which may slow down the algorithm.

As some of the constraints in these Programs (16,17,18) involve a multiplication of variables b and w there is a quadratic term which must be linearized before being input to CPLEX solver. By replacing variables (b,w) with (b′,x), any quadratic terms can be eliminated, and therefore the program can be fed to a linear program solver CPLEX.

By approximating Equation 16 generation with a linear program, this can be fed to a CPLEX solver to indicate whether the corresponding linear program has a feasible solution. Thus, one relaxation approximates Program (16b) with a linear program

$\begin{array}{cc}\max 0\begin{array}{cc}\sum _{s\in S}x\left(s\right)c_{i,j,k}^s+b^{\prime }\left(s\right)d_{i,j,k}^s>0& \forall {\upsilon }_i\in V\\ b^{\prime }\left(s\right)w_{k-1}\le x\left(s\right)\le b^{\prime }\left(s\right)w_k& \forall s\in S\\ \sum _{s\in S}b^{\prime }\left(s\right)=1& \end{array}& 17)\end{array}$

where b′=[b′(s)]_s∈S and x=[x(s)]_s∈S are vectors. Program (17) relaxes Program (16b) because for any feasible solution (b,w) there exists a corresponding feasible solution (b′:=b,x:=bw). If Σ_s∈Sb(s)(c_i,j,k^sw+d_i,j,k^s)>0 in Program (16b), then Σ_s∈Sb(s)wc_i,j,k^s+b(s)d_i,j,k^s>0 and thus, Σ_s∈Sx(s)c_i,j,k^sb′(s)d_i,j,k^s>0 in Program (17), for all υ_i∈V. Next, if w_k−1≦w≦w_k in Program (16b) then for all s∈S, b(s)w_k−1≦b(s)w≦b(s)w_k and thus b′(s)w_k−1≦x(s)≦b′(s)w_k in Program (17). Finally, if Σ_s∈Sb(s)=1 then Σ_s∈Sb′(s)=1. Conversely, a feasible solution (b′,x) may not imply a corresponding feasible solution (b,w). That is, while Σ_s∈Sx(s)c_i,j,k^s+b′(s)d_i,j,k^s>0 in Program (17) implies that Σ_s∈Sb′(s)([x(s)/b′(s)]c_i,j,k^s+d_i,j,k^sk )>0, all the ratios [x(s)/b′(s)],s∈S would need to be equal to some unique w_k−1≦w≦w_k for Σ_s∈Sb′(s)(c_i,j,k^sw+d_i,j,k^sk )>0 to hold.

Because Program (17) relaxes Program (16b), its decision to not prune υ_j from V—a result of finding a feasible solution (b′,x)—in one embodiment, may be too conservative. However, the smaller the wealth interval [w_k−1,w_k], the more accurate Program (17) becomes, that is, the greater the chance that a feasible solution (b′,x) implies a feasible solution (b,w). Thus, for a given feasible solution (b,x), let (b:=b′,w:=w_k−1) be a candidate solution to Program (16b). Clearly Σ_s∈Sb(s)=1 and w_k−1≦w≦w_k. In addition, for all υ_i∈V it holds for C_i^max:=max_s∈S|c_i,j,k^s that

$\begin{array}{c}\left(w_k-w_{k-1}\right)C_i^{\max }+\sum _{s\in S}b\left(s\right)\left(c_{i,j,k}^sw+d_{i,j,k}^s\right)=\sum _{s\in S}b^{\prime }\left(s\right)\left(w_k-w_{k-1}\right)C_i^{\max }+\\ \sum _{s\in S}b^{\prime }\left(s\right)\left(c_{i,j,k}^sw+d_{i,j,k}^s\right)\\ \ge \sum _{s\in S}\left(x\left(s\right)-b^{\prime }\left(s\right)w_{k-1}\right)c_{i,j,k}^s+\\ \sum _{s\in S}b^{\prime }\left(s\right)\left(c_{i,j,k}^sw+d_{i,j,k}^s\right)\\ =\sum _{s\in S}x\left(s\right)c_{i,j,k}^s-b^{\prime }\left(s\right)w_{k-1}c_{i,j,k}^s+\\ b^{\prime }\left(s\right)w_{k-1}c_{i,j,k}^s+b^{\prime }\left(s\right)d_{i,j,k}^s\\ =\sum _{s\in S}x\left(s\right)c_{i,j,k}^s+b^{\prime }\left(s\right)d_{i,j,k}^s>0\end{array}$

and thus, lim_{wk−wk−1→0}Pr[Σ_s∈Sb(s)(c_i,j,k^sw+d_i,j,k^s)>0]=1. Consequently, as w_k−w_k−1→0, the probability that a feasible solution (b′,x) implies a feasible solution (b,w) approaches 1 and the error of approximating Program (16b) with Program (17) approaches 0.

In one embodiment, to speed up the algorithm, the constraint Σ_s∈Sx(s)c_i,j,k^s+b′(s)d_i,j,k^s>0 of Program (17) is tightened by some ε>0. Specifically, it is less likely to find a feasible solution to Program

$\begin{array}{cc}\max 0\begin{array}{cc}\sum _{s\in S}x\left(s\right)c_{i,j,k}^s+b^{\prime }\left(s\right)d_{i,j,k}^s>\varepsilon & \forall {\upsilon }_i\in V\\ b^{\prime }\left(s\right)w_{k-1}\le x\left(s\right)\le b^{\prime }\left(s\right)w_k& \forall s\in S\\ \sum _{s\in S}b^{\prime }\left(s\right)=1& \end{array}& 18)\end{array}$

than to Program (17) and thus, more likely to prune more functions from V, which speeds up the algorithm. However, Program (18) may classify some of the non-dominated functions as dominated ones and hence, the pruning procedure will no longer be error-free. The total error of the algorithm, however, is bounded. In one embodiment, it can be trivially bounded by ε·3·N, where a tunable parameter ε of Program (18) is the error of the pruning procedure, 3 is the number of stages (of the proof by induction) that call the pruning procedure and N is the planning horizon.

Thus, speeding up the algorithm described by equations 16), 17), 18) as solver finds the value functions Vⁿ(b,w) (for the decision epochs n=0,1, . . . ,N) and each value function is represented by a number of bilinear functions. Some of these bilinear functions might be redundant, because they are completely dominated by other bilinear functions and hence, will never be used by the agent when deciding what action to execute. These completely dominated bilinear functions are pruned while the underlying value functions are still represented exactly, but with a reduced number of bilinear functions. This reduces computation time, because the number of bilinear functions needed (e.g., in a worst case) to represent the value function grows exponentially with n.

This methodology scales to larger extensions. For example, there is considered a bigger domain, including 100 different states of the market (e.g., markets of different countries), and considering 5 different actions to invest in markets of different countries. With respect to the algorithm, different values (0.5,1,1.5,2,2.5 ) of the approximation parameter ε (used in Program (18) were tested). Also, the planning horizon was fixed at N=10 and the algorithm is run for each utility function (A),(B),(C),(D),(E) as shown in the plot of utility functions 300 shown in FIG. 3.

FIG. 4A present results 350 plotting “ε” (epsilon) 310 on the x-axis and the runtime 312 (e.g., in seconds on the logarithmic scale) on the y-axes and FIG. 4B is a plot 360 depicting epsilon 310 vs. the solution quality 315 plotted on the y-axes. As can be seen, as shown in FIG. 4B, irrespective of the utility function (A-E) considered in FIG. 3, the algorithm runtime decreases drastically (with only small increases in ε) while the solution quality remains almost constant. For example, for the utility function (C) depicted in plot 350 shown in FIG. 4A, a change of ε from 0.5 to 1.5 caused the reduction of the algorithm runtime by over one order of magnitude (from 149 s to only 12 s) and only 18% (from 9.08 to 7.38 ) decrease of the solution quality as shown in the plot 360 for the utility function (C) of FIG. 4B.

Thus, by employing Risk-Sensitive POMDPs, an extension of POMDPs, in risk domains such as financial planning, the agents are able to maximize the expected utility of their actions. The exact algorithm solves Risk-Sensitive POMDPs, for piecewise linear utility functions by representing the underlying value functions with sets of piecewise bilinear functions—computed exactly using functional value iteration—and pruning the dominated bilinear functions using efficient linear programming approximations of the underlying non-convex bilinear programs.

FIG. 9 illustrates an exemplary hardware configuration of a computing system 400 running and/or implementing the method steps described herein. The hardware configuration preferably has at least one processor or central processing unit (CPU) 411. The CPUs 411 are interconnected via a system bus 412 to a random access memory (RAM) 414, read-only memory (ROM) 416, input/output (I/O) adapter 418 (for connecting peripheral devices such as disk units 421 and tape drives 440 to the bus 412 ), user interface adapter 422 (for connecting a keyboard 424, mouse 426, speaker 428, microphone 432, and/or other user interface device to the bus 412), a communication adapter 434 for connecting the system 400 to a data processing network, the Internet, an Intranet, a local area network (LAN), etc., and a display adapter 436 for connecting the bus 412 to a display device 438 and/or printer 439 (e.g., a digital printer of the like).

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a system, apparatus, or device running an instruction.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a system, apparatus, or device running an instruction. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may run entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which run via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which run on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more operable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be run substantially concurrently, or the blocks may sometimes be run in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

APPENDIX

Induction Base:

Assume n=N. Let Y₀^{N :=B×WN}, I(N):={0} and {dot over (π)}₀^N be an arbitrary policy. Because at decision epoch N the process terminates, it holds for all (b,w)∈Y₀^N that (from Equations (2) and (5)) V_U^N(b,w)=U(w)=E[U(w)]=E[U(w+Σ_n=N^N−1r_n)|{dot over (π)}₀^N,b₀=b])=υ{dot over (π)}₀^N(b,w)=max_i∈I(N)υ{dot over (π)}_i^N(b,w), which proves claim 1. Furthermore, to prove that υ{dot over (π)}₀^Nis piecewise bilinear, let I(N,0):={1, . . . ,K} and W_0,k^N:=[w_k, w_k+1), k∈I(N,0). Clearly, {B×W_0,k^N}_k∈I(N,0) is a finite partitioning of B×Wⁿ and sets W_0,k^N:=k∈I(N,0) are convex. In addition, υ{dot over (π)}₀^N(b,w)=Σ_sb(s)(C_kw+D_k)=C_kw+D_k for all (b,w)∈B×W_0,k^N, k∈I(N,0) and hence, υ{dot over (π)}₀^N(b,w) is linear—thus also piecewise bilinear—over (b,w)∈B×W ^N, which proves claim 2. Finally, claim 3 holds because we constructed υ{dot over (π)}₀^Nwithout even considering the set of functions {υ{dot over (π)}_i′^N+1}_{i′∈I(N+1)} and our choice of {dot over (π)}₀^N was arbitrary. The induction thus holds for n=N.

Induction Step:

Assume now that the induction holds for n+1 . Our goal is to prove that it also holds for n. To this end, recall from Equation (3) that V_Uⁿ(b,w) is calculated by

$\underset{a\in A}{\max }\left\{\sum _{z\in Z}P\left(zb,a\right)V_U^{n+1}\left(T\left(b,a,z\right),w+R\left(b,a\right)\right)\right\}.$

We break this calculation into five stages. First, we calculate V_U,a,zⁿ(b,w):=V_Uⁿ⁺¹(T(b,a,z),w) where V_Uⁿ⁺¹ is represented by {υ{dot over (π)}_iⁿ⁺¹}_i∈I(n+1) from the induction assumption. Next, we derive V_U,a,zⁿ(b,w):=P(z|b,a)V_U,a,zⁿ(b,w) and then V_U,aⁿ(b,w):=Σ_z∈Z(b,w). Finally, we derive V_U,aⁿ(b,w):=V_U,aⁿ(b,w+R(b,a)) and conclude the proof of the induction step by deriving V_Uⁿ(b,w):=max_a∈AV_U,aⁿ(b,w) where V_Uⁿ is represented by {υ{dot over (π)}_iⁿ}_i∈I(n).

Stage 1:

Calculate V_U,a,zⁿ(b,w):=V_Uⁿ⁺¹(T(b,a,z),w).

From the induction assumption, V_Uⁿ⁺¹ is represented by a finite set of functions {υ{dot over (π)}_iⁿ⁺¹}_i∈I(n+1), corresponding to point-based policies {dot over (π)}_i, i∈I(n+1), and each υ{dot over (π)}_iⁿ⁺¹ is piecewise bilinear. We now prove that V_U,a,zⁿ(b,w):=V_Uⁿ⁺¹(T(b,a,z),w) can be represented by a finite set of functions V_a,zⁿ(b,w):={υ_a,z,iⁿ}_i∈I(n+1) derived from a collection of functions {υ{dot over (π)}_iⁿ⁺¹}_i∈I(n+1) and that each function υ_a,z,i ⁿ is piecewise bilinear. To this end, define a finite partitioning {Y_a,z,iⁿ}_i∈I(n+1) of B×Wⁿ⁺¹ where

$\begin{array}{cc}{Y}_{a,z,i}^n:=\left\{\left(b,w\right)\in B\times W^{n+1}\upsilon 〈{\stackrel{.}{\pi }}_i^{n+1}〉\left(T\left(b,a,z\right),w\right)=\underset{i^{\prime }\in I\left(n+1\right)}{\max }\upsilon 〈{\stackrel{.}{\pi }}_{i^{\prime }}^{n+1}〉\left(T\left(b,a,z\right),w\right)\right\}& \left(6\right)\end{array}$

and a finite set of functions V_a,zⁿ={υ_a,z,i^n}_i∈I(n+1) where

υ_a,z,iⁿ(b,w):=υ{dot over (π)}_iⁿ⁺¹(T(b,a,z),w)   (7)

for all (b,w)∈B×Wⁿ⁺¹ . It is then true that for all (b,w)∈B×Wⁿ⁺¹ there exists i∈I(n+1) such that (b,w)∈Y_a,z,iⁿ and υ_a,z,iⁿ(b,w):=υ{dot over (π)}_iⁿ⁺¹(T(b,a,z),w)=max_i′υ{dot over (π)}_iⁿ⁺¹(T(b,a,z),w)=V_U,a,zⁿ⁺¹(T(b,a,z),w)=V_U,a,zⁿ(b,w). Thus, V_U,a,zⁿ(b,w) can be represented by a finite set of functions V_a,zⁿ={υ_a,z,iⁿ}_i∈I(n+1) derived from {υ{dot over (π)}_iⁿ⁺¹}_{i∈I (n+1)}. In addition, each υ_a,z,iⁿ is piecewise bilinear as proven by Lemma 1 in the Appendix.

Finally, notice that if function υ_a,z,iⁿ∈V_a,zⁿ is dominated by other functions υ_a,z,i′ⁿ∈V_a,zⁿ, i.e., if for any (b,w)∈B×Wⁿ⁺¹ there exists i′∈I(n+1),i′≠i such that υ_a,z,iⁿ(b,w)<υ_a,z,i′ⁿ(b,w) then (from definition (6)) Y_a,z,iⁿ=Ø. In such case (to speed up the algorithm) υ_a,z,iⁿ can be pruned from V_a,zⁿ and Y_a,z,iⁿ be removed from {Y_a,z,iⁿ}_i∈I(n+1) as that will not affect the representation of V_U,a,zⁿ. (How to determine if a function υ_a,z,iⁿ is dominated is explained later.) The value functions V_U,a,iⁿ(b,w) can thus be represented by a finite sets of piecewise bilinear functions V_a,zⁿ={υ_a,z,iⁿ}_i∈I(n,a,z) where I(n,a,z)⊂I(n+1) .

Stage 2:

Calculate V_U,a,z ⁿ(b,w):=P(z|b,a)V_U,a,zⁿ(b,w).

Consider the value functions V_U,a,zⁿ(b,w) represented after stage 1 by finite sets of piecewise bilinear functions V_a,zⁿ={υ_a,z,iⁿ}_i∈I(n,a,z). We now demonstrate that the value function V_U,a,zⁿ(b,w):=P(z|b,a)V_U,a,zⁿ(b,w) can be represented by a set of piecewise bilinear functions V_a,zⁿ={ υ_a,z,iⁿ}_i∈I(n,a,z) where

υ _a,z,i ⁿ(b,w):=P(z|b,a)υ_a,z,iⁿ(b,w)   (8)

for all (b,w)∈B×Wⁿ⁺¹. Indeed, since {Y_a,z,iⁿ}_i∈I(n,a,z) is a partitioning of B×Wⁿ⁺¹ (from definition (6)), it holds for all (b,w)∈B×Wⁿ⁺¹ that there exists i∈I(n,a,z) such that (b,w)∈Y_a,z,iⁿ and V_U,a,zⁿ(b,w):=P(z|b,a)V_U,a,zⁿ(b,w)=P(z|b,(b,w)=υ_a,z,iⁿ(b,w) 98_a,z,iⁿ(b,w). Furthermore, each function υ_a,z,iⁿ is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹ because for the existing partitioning {B×W_i,kⁿ⁺¹}_k∈K(n+1,i) of B×Wⁿ⁺¹ it holds that

$\begin{array}{cc}\begin{array}{c}{\stackrel{\_}{\upsilon }}_{a,z,i}^n\left(b,w\right):=P\left(zb,a\right){\upsilon }_{a,z,i}^n\left(b,w\right)\\ =P\left(zb,a\right)\sum _{s\in S}b\left(s\right)\left(c_{a,z,i}^{n,k,s}w+d_{a,z,i}^{n,k,s}\right)\\ =\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,z,i}^{n,k,s}w+{\stackrel{\_}{d}}_{a,z,i}^{n,k,s}\right)\end{array}& \left(9\right)\end{array}$

for all (b,w)∈B×,W_iⁿ⁺¹,k∈I(n+1,i) where c_a,z,i^n,k,s=P(z|b,a)c_a,z,i^n,k,s and d_a,z,i^n,k,s=P(z|b,a)d_a,z,i^n,k,s are constants.

Stage 3:

Calculate V_U,aⁿ(b,w):=Σ_z∈ZV_U,a,zⁿ(b,w).

Consider the value functions V_U,a,zⁿ represented after stage 2 by the sets of piecewise bilinear functions V_a,zⁿ={ υ_a,z,iⁿ}_i∈I(n,a,z). We now show that V_U,a ⁿ can be represented with a finite set of piecewise bilinear functions V_aⁿ={υ_a,i ⁿ}_i∈I(n,a) derived from the sets of functions V_a,zⁿ={ υ_a,z,iⁿ}_i∈I(n,a,z)z∈Z. To this end, let i:=[i(z)]_z∈Z∈I(n,a) denote a vector where i(z)∈I(n,a,z),z∈Z. For each such vector i∈I(n,a) define a set

$\begin{array}{cc}{Y}_{a,i}^n:=\bigcap _{z\in Z}Y_{a,z,i\left(z\right)}^n& \left(10\right)\end{array}$

and a function

$\begin{array}{cc}{\upsilon }_{a,i}^n\left(b,w\right):=\sum _{z\in Z}{\stackrel{\_}{\upsilon }}_{a,z,i\left(z\right)}^n\left(b,w\right)& \left(11\right)\end{array}$

for all (b,w)∈B×Wⁿ⁺¹. To show that V_U,aⁿ can be represented with a set of functions V_aⁿ={υ_a,iⁿ}_i∈I(n,a) we first prove that {Y_a,iⁿ}_i∈(n,a) is a finite partitioning of B×Wⁿ⁺¹. To this end, first observe that Y_a,iⁿ∩Y_a,i′ⁿ=Ø for all i,i′∈I(n,a),i≠i′. Indeed, if i≠i′ then i(z)≠i′(z) for some z∈Z. Thus, if (b,w)∈Y_a,i^n∩Y_a,i′ⁿ then in particular (b,w)∈Y_a,z,i(z)ⁿ∩Y_a,z,i′(z)ⁿ which is impossible because Y_a,z,i(z)ⁿ∩Y_a,z,i′(z) ⁿ≠Ø for i(z)≠i′(z) (from definition (6)). Also, if (b,w)∈B×Wⁿ⁺¹ then for all z∈Z there exists some i(z)∈I(n,a,z) such that (b,w)∈Y_a,z,i(z)ⁿ (from definition (6)). Hence, for the vector i:=[i(z)]_z∈Z∈I(n,a) it must hold that (b,w)∈∩_z∈ZY_a,z,i(z)ⁿ=Y_a,iⁿ.

We then show that V_U,aⁿ can be represented with a set of functions V_aⁿ={υ_a,iⁿ}_i∈I(n,a) as follows: Since {Y_a,iⁿ}_i∈I(n,a) is a partitioning of B×Wⁿ⁺¹, for each (b,w)∈B×Wⁿ⁺¹ there exists i=[i(z)]_z∈Z∈I(n,a) such that (b,w)∈Y_a,iⁿ and V_U,aⁿ(b,w):=Σ_z∈ZV_U,a,zⁿ(b,w)= υ_a,z,i(z)ⁿ(b,w)=υ_a,iⁿ(b,w). In addition, each function υ_a,iⁿ(b,w) is piecewise bilinear as proven by Lemma 2 in the Appendix.

Finally, notice that if function υ_a,iⁿ∈V_aⁿ is dominated by other functions υ_a,i′ⁿ∈V_aⁿ then Y_a,iⁿ=Ø. Precisely, for any (b,w)∈B×Wⁿ⁺¹, if there exists some other function υ_a,i′ⁿ∈V_aⁿ such that υ_a,iⁿ(b,w)<υ_a,i′ⁿ(b,w) then (from definition 11) υ_a,z,i(z)ⁿ(b,w)< υ_a,z,i′(z)(b,w) for some z∈Z and obviously (from definition (9)) υ_a,z,i(z)(b,w)<υ_a,z,i′(z)(b,w) which implies that (from definition (6)) (b,w)∉Y_a,z,i(z) and obviously (from definition (10)), (b,w)∉Y_a,iⁿ. Therefore (to speed up the algorithm), if function υ_a,iⁿ∈V_aⁿ is dominated by other functions υ_a,i′ⁿ∈V_aⁿ then υ_a,iⁿ can be pruned from V_aⁿ and set Y_a,iⁿ be removed from {Y_a,iⁿ}_i∈I(n,a) as that will not affect the representation of V_U,aⁿ.

Stage 4:

Calculate V_U,aⁿ(b,w):=V_U,aⁿ(b,w+R(b,a)).

For notational convenience in this stage (but without the loss of precision), we denote vectors i,k defined in stage 3, as i,k. Recall that Wⁿ is the set of all possible wealth levels at decision epoch n and that Wⁿ⁻¹=[wⁿ⁻¹,wⁿ⁻¹]⊂,[wⁿ, wⁿ]=Wⁿ where wⁿ=wⁿ⁻¹+min_s∈S,a∈AR(s,a) and wⁿ= wⁿ⁻¹+max_s∈S,a∈AR(s,a), for all 1≦n≦N. Hence, we only have to calculate the values V_U,aⁿ(b,w), (b,w)∈B×Wⁿ, from the values V_U,aⁿ(b,w+R(b,a)), (b,w)∈B×Wⁿ⁺¹. To this end, we show how to represent V_U,aⁿ(b,w), (b,w)∈B×Wⁿ with a finite set of piecewise bilinear functions V_aⁿ={ υ_a,iⁿ: B×Wⁿ→R}_i∈I(n,a) derived from the set of piecewise bilinear functions V_aⁿ={υ_a,iⁿ:B×Wⁿ⁺¹→R}_i∈I(n,a) from stage 3. Formally, for each i∈I(n,a) define a set

Y _a,i ⁿ:={(b,w)∈B×Wⁿ

such that

(b,w+R(b,a))∈Y_a,i^n}  (12)

and a function

υ _a,i ⁿ(b,w):=υ_a,iⁿ(b,w+R(b,a)).   (13)

To show that V_U,aⁿ can be represented by { V_aⁿ={ υ_a,iⁿ }_i∈I(n,a) we first need to prove that { Y_a,iⁿ}_i∈I(n,a) is a finite partitioning of B×Wⁿ. Indeed, if (b,w)∈ Y_a,iⁿ∩ Y_a,jⁿ for some i,j∈I(n,a) then (b,w+R(b,a))∈Y_a,iⁿ∩Y_a,jⁿ and thus i=j because {Y_a,iⁿ}_i∈I(n,a) is a partitioning of B×Wⁿ⁺¹ (from stage 3). In addition, for any (b,w)∈B×Wⁿ we have that (b,w+R(b,a))∈B×Wⁿ⁺¹ (because min_s∈S,a∈AR(s,a)≦R(b,a)≦max_s∈S,a∈AR(s,a) and thus, (b,w+R(b,a))∈Y_a,iⁿ for some i∈I(n,a), which implies (from definition (12)) that (b,w)∈ Y_a,iⁿ.

We then show that V_U,aⁿ(b,w) can be represented for all (b,w)∈B×Wⁿ with the set of functions V_a,iⁿ={ υ_a,iⁿ}_i∈I(n,a) as follows: Since { Y_a,iⁿ}_i∈I(n,a) is a finite partitioning of B×Wⁿ, for all (b,w)∈B×Wⁿ there exists i∈I(n,a) such that (b,w)∈ B_a,iⁿ× W_a,iⁿ and V_U,aⁿ(b,w):=V_U,aⁿ(b,w +R(b,a))=υ_a,iⁿ(b,w+R(b,a))= υ_a,iⁿ(b,w). In addition, each function υ_a,iⁿ(b,w)∈ V_aⁿ is piecewise bilinear over (b,w)∈B×Wⁿ and can be derived from υ_a,iⁿ∈V_aⁿ, as shown in Lemma (3) in the Appendix.

Stage 5:

Calculate V_U(b,w):=max_a∈AV_U,aⁿ(b,w).

Consider the value functions V_U,aⁿ represented after stage 4 by the set of piecewise bilinear functions V_aⁿ={ υ_a,iⁿ}_i∈I(n,a). To conclude the proof of the induction step, we show how to represent V_Uⁿ with a finite set of piecewise bilinear functions Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n) derived from functions from sets V_aⁿa∈A. To this end, let I(n):={(a,i)|a∈A,i=[i(z)]_z∈Z∈I(n,a)}. For each pair (a,i)∈I(n) then define a set

$\begin{array}{cc}{Y}_{\left(a,1\right)}^n:=\left\{\left(b,w\right)\in B\times W^n{\stackrel{\_}{v}}_{a,1}^n\left(b,w\right)=\underset{\left(a^{\prime },i^{\prime }\right)\in I\left(n\right)}{\max }{\stackrel{\_}{\upsilon }}_{a^{\prime },i^{\prime }}^n\left(b,w\right)\right\}& \left(14\right)\end{array}$

and a point based policy {dot over (π)}_(a,i)ⁿ according to which the agent first executes action a∈A and then, depending on the observation z∈Z received, follows the policy {dot over (π)}_i(z)ⁿ⁺¹ given by the induction assumption.

Clearly, {Y_(a,i)ⁿ}_(a,z)∈I(n) is a finite partitioning of B×Wⁿ. Thus, for all (b,w)∈B×Wⁿ there exists some (a,i)∈I(n) such that (b,w)∈Y_(a,i)ⁿ and

$\begin{array}{cc}{V}_U^n\left(b,w\right):=\underset{\left(a,i\right)\in I\left(n\right)}{\max }{\stackrel{\_}{\upsilon }}_{a,i}^n\left(b,w\right)\equiv v〈{\stackrel{.}{\pi }}_{\left(a,i\right)}^n〉\left(b,w\right)& \left(15\right)\end{array}$

(the last equality follows directly from definitions (13) (11) (8) (7)). Therefore, V_Uⁿ can indeed be represented by a finite set of piecewise bilinear functions Vⁿ={υ{dot over (π)}_(a,i)ⁿ}_(a,i)∈I(n)={ υ_a,iⁿ}_(a,i)∈I(n) derived (through stages 1,2,3,4,5) from functions {υ{dot over (π)}_i′ⁿ⁺¹}_{i′∈I(n+1)}, which proves claims 1, 2 and 3 of the induction step and the whole proof by induction.

Finally, notice that if a function υ{dot over (π)}_(a,i)ⁿ∈Vⁿ is dominated by other functions υ{dot over (π)}_(a′,i′)ⁿ∈Vⁿ, i.e., if for all (b,w)∈B×Wⁿ there exists some υ{dot over (υ)}_(a′,i′)ⁿ∈Vⁿ such that υ{dot over (π)}_(a,i)ⁿ(b,w)<υ{dot over (π)}_(a′,i′)ⁿ(b,w) then Y_(a,i)ⁿ=Ø. In such case, (to speed up the algorithm) υ{dot over (υ)}_(a,i)ⁿ can be pruned from Vⁿ and Y_(a,i)ⁿ be removed from {Y_,(a,i)ⁿ}_(a,i)∈I(n) as that will not affect the representation of V_Uⁿ.

Lemma 1

Function υ_a,z,iⁿ:=υ{dot over (π)}_iⁿ⁺¹(T(b,a,z),w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹.

Proof.

From induction assumption, υ{dot over (π)}_iⁿ⁺¹(b,w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹, i.e., there exists a finite partitioning {B×W_i,kⁿ⁺¹}_{k∈I(n +1,i)} of B×Wⁿ⁺¹ such that W_i,kⁿ⁺¹ is a convex set and υ{dot over (π)}_iⁿ⁺¹(b,w)=Σ_s∈Sb(s)(c_i,k,sⁿ⁺¹w+d_i,k,sⁿ⁺¹) for all (b, w)∈B×W_i,kⁿ⁺¹,k∈I(n+1,i). We now prove that υ_a,z,iⁿ(b,w):=υ{dot over (π)}_iⁿ⁺¹(T(b,a,z),w) too is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹ for the partitioning {B×W_i,kⁿ⁺¹}_k∈I(n+1,i) of B×Wⁿ⁺¹. To this end, for each s∈S distinguish a belief state b_s∈B such that b_s(s)=1. It then holds for all (b,w)∈B×W_iⁿ⁺¹,k∈I(n+1,i) that

$\begin{array}{cc}\begin{array}{c}{\upsilon }_{a,z,i}^n\left(b,w\right):=\upsilon 〈{\stackrel{.}{\pi }}_i^{n+1}〉\left(T\left(b,a,z\right),w\right)\\ =\sum _{s^{\prime }\in S}\left[T\left(b,a,z\right)\left(s^{\prime }\right)\right]\left(c_{i,k,s^{\prime }}^{n+1}w+d_{i,k,s^{\prime }}^{n+1}\right)\\ =\sum _{s^{\prime }\in S}\sum _{s\in S}b\left(s\right)\left[\left(b_s,a,z\right)\left(s^{\prime }\right)\right]\left(c_{i,k,s^{\prime }}^{n+1}w+d_{i,k,s^{\prime }}^{n+1}\right)\\ =\sum _{s\in S}b\left(s\right)\sum _{s^{\prime }\in S}P\left(s^{\prime }s,a\right)O\left(za,s^{\prime }\right)\left(c_{i,k,s^{\prime }}^{n+1}w+d_{i,k,s^{\prime }}^{n+1}\right)\\ =\sum _{s\in S}b\left(s\right)\left(c_{a,z,i}^{n,k,s}w+d_{a,z,i}^{n,k,s}\right)\end{array}& \left(19\right)\end{array}$

for constants c_a,z,i^n,k,sΣ_s′∈SP(s′|s,a)O(z|a,s′)c_i,k,s′ⁿ⁺¹ and d_a,z,i^n,k,sΣ_s′∈SP(s′|s,a)O(z|a,s′)d_i,k,s′ⁿ⁺¹. Consequently, function υ_a,z,iⁿ(b,w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹ which proves the Lemma.

Lemma 2

Function υ_a,iⁿ(b,w):=Σ_z∈Zυ_a,z,i(z)ⁿ(b,w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹.

Proof.

After stage 2 it holds for all z∈Z that υ_a,z,i(z)ⁿ(b,w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹, i.e., there exist a partitioning {B×W_i(z),k ⁿ⁺¹}_{k␣I(n+1,i(z))} of B×Wⁿ⁺¹ such that W_i(z),kⁿ⁺¹ is a convex set and υ_a,z,i(z)ⁿ(b,w)=Σ_s∈Sb(s)( c_a,z,i(z)^n,k,sw+ d_a,z,i(z)^n,k,s)for all (b,w)∈B×W_i(z),kⁿ⁺¹k∈I(n+1,i(z)). To prove that υ_a,iⁿ(b,w):=Σ_z∈Zυ_a,z,i(z)ⁿ(b,w) too is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹ we represent υ_a,iⁿ with the set of bilinear functions {υ_a,i,kⁿ}_{k∈I(n,a,i). Precisely, let k:=[k(z)]z∈Z}∈I(n,a,i) denote a vector where k(z)∈I(n+1,i(z)). For each vector k∈I(n,a,i) we define a set

$\begin{array}{cc}{W}_{a,i,k}^{n+1}:=\bigcap _{z\in Z}W_{i\left(z\right),k\left(z\right)}^{n+1}& \left(20\right)\end{array}$

and a bilinear function

$\begin{array}{cc}{\upsilon }_{a,i,k}^n\left(b,w\right):=\sum _{s\in S}b\left(s\right)\left(c_{a,i}^{n,k,s}w+d_{a,i}^{n,k,s}\right)& \left(21\right)\end{array}$

for all (b,w)∈B×Wⁿ⁺¹ and constants c_a,i^n,k,s:=Σ_z∈Zc_a,z,i(z)^n,k(z),s, d_a,i^n,k,s:=Σ_z∈Zd_a,z,i(z)^n,k(z),s. To show that υ_a,iⁿ(b,w) can be represented by {υ_a,i,kⁿ(b,w)}_k∈I(n,a,i) over all (b,w)∈B×Wⁿ⁺¹ we first prove that {B×W_a,i,kⁿ⁺¹}_k∈I(n,a,i) is a finite partitioning of B×Wⁿ⁺¹. To this end, first observe that W_a,i,k ⁿ⁺¹∩W_a,i,k′ⁿ⁺¹=Ø for any k,k′∈I(n,a,i),k≠k′. Indeed, if k≠k′ then k(z)≠k′(z) for some z∈Z. Hence, if w∈W_a,i,kⁿ⁺¹∩W_a,i,k′ⁿ⁺¹ then in particular w∈W_i(z),k(z)ⁿ⁺¹∩W_i(z),k′(z)ⁿ⁺¹ which is cannot be true as W_i(z),k(z)ⁿ⁺¹∩W_i(z),k′(z)ⁿ⁺¹=Ø for k(z)≠k′(z) (from claim 2 of the induction assumption). Also, observe that for any w∈Wⁿ⁺¹ there must exist k∈I(n,a,i) such that w∈W_a,i,kⁿ⁺¹, because for all z∈Z, there exists k(z)∈I(n+1,i(z)) such that w∈W_i(z),k(z)ⁿ⁺¹ (since {W_i(z),k(z)}_{k(z)∈(n+1,i (z))} is a partitioning of Wⁿ⁺¹, from claim 2 of the induction assumption). Thus, vector k:=[k(z)]_z∈Z∈I(n,a,i) such that w∈∩_z∈ZW_a,i(z),k(z)ⁿ⁺¹=W_a,i,kⁿ⁺¹ truly exists. Consequently, {W_a,i,kⁿ⁺¹}_k∈I(n,a,i) is a finite partitioning of Wⁿ⁺¹ and {B×W_a,i,kⁿ⁺¹}_k∈I(n,a,i) a finite partitioning of B×Wⁿ⁺¹.

We can therefore prove that functions {υ_a,i,kⁿ}_k∈I(n,a,i) represent υ_a,iⁿ(b,w) over all (b,w)∈B×Wⁿ⁺¹ as follows: For each (b,w)∈B×Wⁿ⁺¹ there exists k∈I(n,a,i) such that (b,w)∈B×W_a,i,kⁿ⁺¹. Hence, (from definition (20)) (b,w)∈B×W_i(z),k(z)ⁿ⁺¹ and thus, (from definition (9)) υ_a,z,iⁿ(b,w)=Σ_s∈Sb(s)( c_a,z,i(z)ⁿ⁺¹w+ d_a,z,i(z))ⁿ⁺¹. We can then easily prove that υ_a,iⁿ(b,w):=Σ_z∈Zυ_a,z,i(z)ⁿ(b,w)=Σ_z∈ZΣ_s∈Sb(s)( c_a,z,i(z)^n,k(z),sw+ d_a,z,i(z)^n,k(z),s)=Σ_s∈S(c_a,i^n,k,sw+d_a,i^n,k,s)=υ_a,i,k ⁿ(b,w). Finally, each set W_a,i,kⁿ⁺¹ is convex because (from definition (20)) it is an intersection of convex sets W_i(z),k(z)ⁿ⁺¹, z∈Z.

Lemma 3

Function υ_a,iⁿ(b,w):=υ_a,iⁿ(b,w+R(b,a)) is piecewise bilinear over (b,w)∈B×Wⁿ.

Proof.

After stage 3 it is true for all i∈I(n,a) that υ_a,iⁿ(b,w) is piecewise bilinear over (b,w)∈B×Wⁿ⁺¹, i.e., there exist a partitioning {B×W_a,i,kⁿ⁺¹}_k∈I(n,a,i) of B×Wⁿ⁺¹ such that W_a,i,kⁿ⁺¹ is convex and υ_a,iⁿ(b,w)=υa,i,kⁿ(b,w)=Σ_s∈Sb(s)(c_a,i^n,k,sw+d_a,i^n,k,s) for all (b,w)∈B×W_a,i,kⁿ⁺¹, for all k∈I(n,a,i). To prove that υ_a,iⁿ(b,w):=υ_a,iⁿ(b,w+R(b,a)) is piecewise bilinear over (b,w)∈B×Wⁿ we represent υ_a,iⁿ with a set of bilinear functions { υ_a,i,kⁿ}_{k∈Ī(n,a,i)}. To this end, first, for each k∈I(n,a,i),s∈S define a set

W _a,i,k ^n,s :={w∈W ⁿ |w+R(s,a)∈W_a,i,kⁿ⁺¹}  (22)

Now, let k:=[k(s)]_s∈S denote a vector where k(s)∈I(n,a,i). Ī(n,a,i) is a set of all such vectors k. For each vector k∈(n,a,i) then define a set

$\begin{array}{cc}{\stackrel{\_}{W}}_{a,i,k}^n:=\bigcap _{s\in S}{\stackrel{\_}{W}}_{a,i,k\left(s\right)}^{n,s}& \left(23\right)\end{array}$

and a bilinear function

$\begin{array}{cc}{\stackrel{\_}{\upsilon }}_{a,i,k}^n\left(b,w\right):=\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,i}^{n,k\left(s\right),s}w+{\stackrel{\_}{d}}_{a,i}^{n,k\left(s\right),s}\right)& \left(24\right)\end{array}$

for all (b,w)∈B×Wⁿ where c_a,i^n,k(s),s:=c_a,i^n,k(s),s and d_a,i^n,k(s),s:=d_a,i^n,k(s),s+c_a,i^n,k(s),sR(s,a) are constants. To show that υ_a,iⁿ can be represented by { υ_a,i,kⁿ}_{k∈Ī(n,a,i)} we first prove that { W_a,i,kⁿ}_{k∈Ī(n,a,i)} is a finite partitioning of Wⁿ. Indeed, for any k,k′∈Ī(n,a,i) if w∈ W_a,i,kⁿ∩ W_a,i,k′ⁿ then (from definition (23)) for all s∈S, w∈ W_a,i,k(s)^n,s∩ W_a,i,k′(s)^n,s and thus (from definition (22)) w+R(s,a)∈W_a,i,k(s)ⁿ⁺¹∩W_a,i,k′(s)ⁿ⁺¹ for all s∈S, which can only hold if k=k′ (because {W_a,i,k(s)ⁿ⁺¹}_{k(s)∈I(n,a,i)} is a partitioning of Wⁿ⁺¹). In addition, for any w∈Wⁿ,s∈S it holds that w+R(s,a)∈Wⁿ⁺¹ and thus, there must exists some k(s)∈I(n,a,i) such that w+R(s,a)∈W_a,i,k(s)ⁿ⁺¹. Therefore (from definition (22)) w∈ W_a,i,k(s)ⁿ for all s∈S and thus (from definition (23)) w∈ W_a,i,kⁿ. We have therefore proven that { W_a,i,kⁿ}_{k∈Ī(n,a,i)} is a finite partitioning of Wⁿ and that {B× W_a,i,kⁿ}_{k∈Ī(n,a,i)} is a finite partitioning of B×Wⁿ.

We then show that functions { υ_a,i,kⁿ}_{k∈Ī(n,a,i)} represent υ_a,iⁿ(b,w) over all (b,w)∈B×Wⁿ as follows: For each (b,w)∈B×Wⁿ there must exist k∈Ī(n,a,i) such that (b,w)∈B× W_a,i,kⁿ and (b,w+R(s,a))∈B× W_a,i,k(s)ⁿ⁺¹∀s∈S, for which it holds that^{1 1}Recall that for each s∈S we distinguish b_s∈B such that b _s(s)=1.

$\begin{array}{cc}\begin{array}{c}{\stackrel{\_}{\upsilon }}_{a,}^n\left(b,w\right):={\upsilon }_{a,i}^n\left(b,w+R\left(b,a\right)\right)\\ =\sum _{s\in S}b\left(s\right){\upsilon }_{a,i}^n\left(b_s,w+R\left(b_s,a\right)\right)\\ =\sum _{s\in S}b\left(s\right)\sum _{s^{\prime }\in S}b_s\left(s^{\prime }\right)\left(c_{a,i}^{n,k\left(s\right),s^{\prime }}\left(w+R\left(s,a\right)\right)+d_{a,i}^{n,k\left(s\right),s^{\prime }}\right)\\ =\sum _{s\in S}b\left(s\right)\left(c_{a,i}^{n,k\left(s\right),s}w+c_{a,i}^{n,k\left(s\right),s}R\left(s,a\right)+d_{a,i}^{n,k\left(s\right),s}\right)\\ =\sum _{s\in S}b\left(s\right)\left({\stackrel{\_}{c}}_{a,i}^{n,k\left(s\right),s}w+{\stackrel{\_}{d}}_{a,i}^{n,k\left(s\right),s}\right)\\ ={\stackrel{\_}{\upsilon }}_{a,i,k}^n\left(b,w\right)\end{array}& \left(25\right)\end{array}$

Finally, each set W_a,i,kⁿ is convex because it is an intersection of convex sets W_a,i,k(s)^n,s, s∈S (translation of a convex set W_a,i,k(s)ⁿ⁺¹ by a vector R(s,a) results in a convex set).

Claims

1. A method for determining an investment strategy for a risk-sensitive user comprising: modeling an user's attitude towards risk as one or more utility functions, said utility functions, said utility function transforming a wealth of said user into a utility value;generating a risk-sensitive Partially Observable-Markov Decision Process (PO-MDP) based on said one or more utility functions; and,implementing Functional Value Iteration for solving said risk sensitive PO-MDP,said solution determining an action or policy calculated to maximize an expected total utility of an agent's actions at a particular point in time acting in a partially observable environment.

2. The method as in claim 1, wherein said generating said risk-sensitive PO-MDP comprises: generating an expected utility function VUn(b,w) for 0≦n≦N, b∈B, w∈Wn where Wn denotes the set of all possible user wealth levels in decision epoch n; and,maximizing said expected utility function VUn(b,w) for a user when commencing action a∈A, where A is a set of Actions, in decision period n in a belief state b with a wealth level w.

3. The method as in claim 2, further comprising: receiving incomplete information about a current state s∈S of the process; and,representing a belief state b as a current probability distribution b(s) over states s∈S.

4. The method as in claim 3, wherein said expected utility function VUn(b,w) for executing action a is governed according to:

5. The method as in claim 4, further comprising: iteratively constructing a finite partitioning of a B×W search space into regions where said value functions are represented with point based policies; anddetermining from said regions an action.

6. The method as in claim 5, further comprising, at each iteration: representing VUn+1(b,w) using a finite set of bilinear functions γn+1; and,constructing, from said set of bilinear functions from γn+1, a set of bilinear functions γn that jointly represent VU n(b,w), wherein at an end of each said iteration,determining from said set of bilinear functions γn what action a∈A said user should execute in decision epoch n∈[0, 1, . . . , N], with wealth level w∈[wmin, wmax], given an inventor belief state b(s), for all s∈S.

7. The method as in claim 6, further comprising: determining whether a bilinear function of said bilinear functions is jointly dominated by other functions; and,pruning from γn those bilinear functions that are completely dominated by other bilinear functions.

8. The method as in claim 7, wherein said determining whether a function is jointly dominated comprises: splitting said functions of into functions defined over common wealth interval wk−1≦w≦wk; and,determining if a feasible solution (b,w) exists for 1≦k≦K according to a first program having quadratic terms; and,linearizing said first program to obtain a second program having linear teens.

9. The method as in claim 8, wherein said first program is governed according to:

10. The method as in claim 9, further comprising; tightening a constraint Σs∈Sx(s)ci,j,ks+b′(s)di,j,ks>0 by a value ε∈>0 wherein said second program is governed according to:

11. A system for determining an investment strategy for a risk-sensitive user comprising: a memory;a processor in communications with the memory, wherein the system performs a method comprising: modeling an user's attitude towards risk as one or more utility functions, said utility functions said utility function transforming a wealth of said user into a utility value;generating a risk-sensitive Partially Observable-Markov Decision Process (PO-MDP) based on said one or more utility functions; and,implementing Functional Value Iteration for solving said risk sensitive PO-MDP,said solution determining an action or policy calculated to maximize an expected total utility of an agent's actions at a particular point in time acting in a partially observable environment.

12. The system as in claim 11, wherein said generating said risk-sensitive PO-MDP comprises: generating an expected utility function VU n(b,w) for 0≦n≦N, b∈B, w∈Wn where Wn denotes the set of all possible user wealth levels in decision epoch n; and,maximizing said expected utility function VUn(b,w) for a user when commencing action a∈A, where A is a set of Actions, in decision period n in a belief state b with a wealth level w.

13. The system as in claim 12, further comprising: receiving incomplete information about a current state s∈S of the process; and,representing a belief state b as a current probability distribution b(s) over states s∈S.

14. The system as in claim 13, wherein said expected utility function VUn(b,w) for executing action a is governed according to:

15. The system as in claim 14, wherein said system further performs: iteratively constructing a finite partitioning of a B×W search space into regions where said value functions are represented with point based policies; anddetermining from said regions an action.

16. The system as in claim 15, further comprising, at each iteration of said Functional Value Iteration: representing VUn+1(b,w) using a finite set of bilinear functions γn+1; and,constructing, from said set of bilinear functions from γn+1, a set of bilinear functions γn that jointly represent VUn(b,w), wherein at an end of each said iteration,determining what action (policy) a∈A should said user execute in decision epoch n∈[0, 1, . . . , N], with wealth level w∈[wmin, wmax], given an inventor belief state b(s), for all s∈S.

17. The system as in claim 16, further comprising: determining whether a bilinear function of said bilinear functions is jointly dominated by other functions; and,pruning from γn those bilinear functions that are completely dominated by other bilinear functions.

18. The system as in claim 17, wherein said determining whether a function is jointly dominated comprises: splitting said functions of into functions defined over common wealth interval wk−1≦w≦wk; and,determining if a feasible solution (b,w) exists for 1≦k≦K according to a first program having quadratic terms; and,linearizing said first program to obtain a second program having linear terms.

19. The system as in claim 18, wherein said first program is governed according to:

20. The system as in claim 19, further comprising; tightening a constraint Σs∈Sx(s)ci,j,ks+b′(s)di,j,ks>0 by a value ε>0 wherein said second program is governed according to:

21. A computer program product for determining an investment strategy for a risk-sensitive user, the computer program product comprising: a storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising: modeling an user's attitude towards risk as one or more utility functions, said utility functions said utility function transforming a wealth of said user into a utility value;generating a risk-sensitive Partially Observable-Markov Decision Process (PO-MDP) based on said one or more utility functions; and,implementing Functional Value Iteration for solving said risk sensitive PO-MDP,said solution determining an action or policy calculated to maximize an expected total utility of an agent's actions at a particular point in time acting in a partially observable environment.

22. The computer program product as in claim 21, wherein said generating said risk-sensitive PO-MDP comprises: generating an expected utility function VUn(b,w) for 0≦n≦N, b∈B, w∈Wn where Wn denotes the set of all possible user wealth levels in decision epoch n; and,maximizing said expected utility function VU n(b,w) for a user when commencing action a∈A, where A is a set of Actions, in decision period n in a belief state b with a wealth level w.

23. The computer program product as in claim 22, wherein said expected utility function VUn(b,w) for executing action a is governed according to:

24. The computer program product as in claim 23, further comprising: iteratively constructing a finite partitioning of a B×W search space into regions where said value functions are represented with point based policies; anddetermining from said regions an action.

25. The computer program product as in claim 5, further comprising, at each iteration: representing VUn+1(b,w) using a finite set of bilinear functions γn+1; and,constructing, from said set of bilinear functions from γn+1, a set of bilinear functions γn that jointly represent VUn(b,w), wherein at an end of each said iteration,determining from said set of bilinear functions γn what action a∈A should said user execute in decision epoch n∈[0, 1, . . . , N], with wealth level w∈[wmin, wmax], given an inventor belief state b(s), for all s∈S.

26. The computer program product as in claim 25, further comprising: determining whether a bilinear function of said bilinear functions is jointly dominated by other functions; and,pruning from γn those bilinear functions that are completely dominated by other bilinear functions.