Patent Application
20190124667
The field of the invention is that of multi-access telecommunication systems offering a plurality of transmission resources. The invention relates to the management of transmission resources and is more particularly concerned with a scheduling implemented in two decoupled steps with first a scheduling in the time domain to select a group of data flows based on needs in terms of quality of service, typically the more urgent data flows, and then a scheduling in the frequency domain taking the state of the transmission channel into account to allocate the radio resources to the data flows of the selected group.
5th generation mobile networks will be characterised by the presence of multiple services the needs of which will be extremely heterogeneous. For example, the services of extreme mobile broadband require very high data rates, ultra-reliable low latency applications require very low latencies and very low error rates, and massive machine-type communication services will impose the presence of a very significant density of user connections. These different needs are often contradictory and the joined optimisation of these needs is a difficult problem to solve.
The OFDMA (Orthogonal Frequency Division Multiple Access) technique allows a fine-grained allocation of the radio resource where each resource unit RB (Resource Block) unit can be allocated and modulated adaptatively to make use and take advantage of the frequency diversities of each user. To fulfil the heterogeneous needs in terms of Quality of Service (QoS) of the different services of a 5th generation network, a flexible radio resource management (RRM) is necessary.
The management of the radio resource makes use of a packet scheduler which supports sharing the radio resource between the different active users at each transmission time interval. This scheduler has a set of scheduling rules used to control the parameters such as transmission power, coding and modulation scheme and the allocated bandwidth. The aim is to use as efficiently as possible the radio resources to increase the spectral efficiency, while ensuring to the users the service level required by each traffic request.
The four following types of scheduling rules can be distinguished.
None of these rules can be considered as the most appropriate solution for all the network conditions and all the QoS needs. To fulfil delay requirements, a scheduling with two decoupled steps has been proposed with first a scheduling in the time domain to select a group of packets based on needs in terms of quality of service, typically the most urgent packets, and then a scheduling in the frequency domain taking the state of the transmission channel into consideration to allocate the radio resources to the packets of the selected group. Such a scheduling is for example known from patents EP 2 148 478 B1 and EP 2 936 908 B1. But these scheduling solutions do not enable multiple traffic categories having different needs in terms of QoS to be differentiated.
The purpose of the invention is to provide a method for scheduling packets belonging to a plurality of packet categories in a multi-access telecommunication system sharing a plurality of transmission resources, capable of treating heterogeneous data flows while being able to offer optimum performance in terms of fulfiling QoS requirements.
To that end, the invention provides a method which comprises, for a state of the multi-access telecommunication system, determining, by iterations of a reinforcement learning, a resource allocation plan maximising a reward sum. Each iteration of the reinforcement learning comprises:
Some preferred but not limiting aspects of this method are the following ones:
Further aspects, purposes, advantages and characteristics of the invention will be better apparent upon reading the following detailed description of preferred embodiments thereof, given by way of non-limiting examples, and made in reference to the appended drawings in which:
The invention addresses the scope of a multi-access telecommunication system where a plurality of users and/or applications have access to common transmission resources, and in which, for each access, data are transmitted as packets. A typical example of use of the invention is that of an OFDMA system in which the available bandwidth is divided into NRB resource units RB (Resource Block) where a resource unit RB represents the minimum amount of frequency resources that a base station can allocate at each transmission time interval (TTI).
Within the scope of the invention, the telecommunication system offers its transmission resources to P categories of data flows (in what follows, a data flow is associated with a user). TC={1 . . . P} thus denotes all the data flow categories. Each category p∈TC is characterised by a set of active users UEP={1 . . . Ip} (i.e. a set of data flows) and a set of specific objectives and requirements in terms of QoS, noted Op={1 . . . NO
In order to efficiently schedule heterogeneous data flow categories having different priorities, the invention makes use of a planner configured to, at each transmission time interval, select a resource allocation plan and make an allocation of the NRB transmission resources to the data flows in accordance with the selected resource allocation plan. In a favoured embodiment represented in
At a TTI transmission interval t, the telecommunication system has a current state s[t] and each data flow category p∈TC has a current state sp[t]. The scheduling in the time domain is made by a time classifier CT configured to make the selection of a hierarchisation rule and the hierarchisation of the data flow categories according to the selected hierarchisation rule into at least one priority category and one secondary category. The time classifier CT thus hierarchises data flow categories as a function of their current states and iteratively forwards them to a frequency classifier CF, from the highest priority category to the lowest priority category, to define the scheduling in the frequency domain.
A possible hierarchisation strategy consists in determining the scheduling of the data flows according to a priority level which corresponds to each flow, as defined for example in table 6.1.7 of 3GPP Technical Specification Group Services and System Aspects, “TS 23.203, Policy and charging control architecture”, (Release 14), V14.4.0, June 2017. Another possible strategy is the hierarchisation of the data flows giving priority to the GBR (guaranteed bit rate) flows, versus the non-GBR (not guaranteed bit rate) flows.
In the example of
The scheduling in the frequency domain is made by the frequency classifier CF which makes use as an input the state of the users {sp,1[t], . . . , sp,I
The frequency classifier FC thus selects the most appropriate scheduling rule up from NR possible rules (up∈U={0, 1, . . . , NR}) to improve QoS of the data flows of the priority category p. Here, uk=0 indicates that the category k is not scheduled because of the absence of remaining resources. Examples of scheduling rules have been given in the introductory part.
The application of the most appropriate scheduling rule up provides for each user ip∈UEp of the selected category p his/her priority Ψu
Within the scope of the invention, the planner is configured to query a look up table LUT in order to identify, from the current state s[t] of the multi-access telecommunication system, the resource allocation plan to be implemented. Within the scope of the favoured embodiment, one or both of the time CT and frequency CF classifiers is/are configured to perform, at each transmission time interval TTI, the querying of the look up table LUT to identify, from the current state s[t] of the multi-access telecommunication system, respectively, the hierarchisation rule and/or the first scheduling rule up (and optionally the scheduling rule(s) applicable to the secondary category(ies) being resource-allocated) to be selected.
According to a first aspect, the invention is concerned with creating the look up table. More particularly, it relates to a method for determining a resource allocation to packets belonging to a plurality of data flow categories in a multi-access telecommunication system sharing a plurality of transmission resources, this method comprising, for a state of the multi-access telecommunication system, determining, by iterations of a reinforcement learning, an allocation of resource maximising a reward sum.
Each iteration of the reinforcement learning comprises:
Thus, to make it possible to fulfil requirements in terms of QoS of heterogeneous data flow categories, the invention uses a reinforcement learning to learn the optimum resource allocation plan to be implemented by the planner. Within the scope of the favoured embodiment, the reinforcement learning makes it possible to determine the hierarchisation rule and/or optimum scheduling rules to be implemented respectively by the time classifier which dynamically hierarchises the different data flow categories and by the frequency classifier which selects the most appropriate scheduling rule to allocate transmission resources to the activated data flow category (i.e. first the priority category and then the secondary categories according to the hierarchisation made by the time classifier).
In one embodiment, both time and frequency classifiers make use of a look up table from a reinforcement learning.
In another embodiment, only the frequency classifier makes use of a look up table from a reinforcement learning. This learning enables the most appropriate strategy to allocate transmission resources to the different data flow categories to be learned and thus the scheduling rule to be applied to a given category to be determined in order to optimise QoS. In such a case, the time classifier can be relatively simple, and for example implement a Round-Robin type strategy or a strategy based on fixed priorities for the data flow categories (such that a first category is only treated by the frequency classifier when the higher-priority categories have no data to transmit).
In yet another embodiment, only the time classifier makes use of a look up table from a reinforcement learning. This learning enables, for a current state of the system s[t], the most appropriate hierarchisation rule of the data flow categories to be determined. In such a case, the frequency classifier can be relatively simple, and for example implement the MCI strategy or EDF strategy.
The implementation of the reinforcement learning within the scope of the favoured embodiment in which both classifiers make use of a look up table from such a learning is described in the following.
Generally, the object of the reinforcement learning is to learn, from experiments, what should be done in different situations, so as to optimise a quantitative reward over time. At each instant t, a controller observes the full state of the system s and performs a given action a. At instant t+1, the controller observes a new state s′ and receives a reward r which evaluates the optimality of the previous action. The controller passes through a set of transitions (i.e. from one state to another) and explores all the possible state-action pairs until it determines the decisional behaviour (called strategy or policy, and which is a function associating the current state to the action to be executed) being optimum in that it maximises the reward sum over time.
Let S be the state space of the system, and s=[s1, . . . , sP]∈S the current state of the system, which describes the state of each of the data flow categories. The state sp of the category p is defined by the state of each of its users, sp=[sp,1[t], . . . , sp,I
The QoS requirements of the category p towards these indicators KPI can be described by the vector
with
A=ACT×ACF denotes the set of possible actions of the controller of the reinforcement learning, with ACT the set of actions corresponding to the time classifier and ACF the set of actions corresponding to the frequency classifier. More specifically, ACT={ACT,1, . . . , ACT,N} is a set with a size N=P!, where each action corresponds to a possible hierarchisation of the P data flow categories. And ACF={1, . . . , NR} is a set of scheduling rules with a size NR, where the action i∈ACF corresponds to the selection of the ith scheduling rule. Thus, a={aCT,aCF}∈A is the pair of actions consisting of the actions made by each of the classifiers.
The reward function R calculates the optimality to apply the action a from a state s. It takes into consideration, for each of the data flows of each of the categories and for each performance indicator, the deviation between the instantaneous indicator of the data flow and the threshold representative of a quality of service requirement relating to the indicator for the category to which the data flow belongs.
This reward function for the time classifier can be determined as R(s,aCS,aRS)=Σp=1Prp(sp,aCS,up), where rp(sp,aCS,up) is the reward received at the frequency classifier when the scheduling rule up is used for the category p with the state sp, given the selection of the hierarchisation rule aCS. Thus, the reward which is received at the time classifier corresponds to the sum of all the rewards of the frequency classifier. At the frequency classifier, the reward function can be determined as
namely as being the reward sum for each user ip∈UEp and for each indicator KPI o∈Op.
According to the invention, the reinforcement learning can make a compromise between exploitation and exploration, where exploitation consists in remaking the actions which, depending on the acquired experience, will maximise the cumulated reward and where exploration consists in looking over (state, action) couples insearching for a larger cumulated reward, with the risk of assuming sometimes a sub-optimal behaviour. More particularly, the reinforcement learning retains a compromise between exploration and exploitation which consists in following the current optimal policy most of the time, while choosing a random action more or less regularly.
Thus, the reinforcement learning according to the invention comprises at each of the iterations:
In a first possible embodiment, the mechanism called ε-greedy is adopted, according to which Pa=1−ε, ε being positive and strictly lower than 1, and
where K is equal to N or NR depending on whether the hierarchisation rule or the scheduling rule is considered. Alternative solutions to this mechanism exist (for example the Boltzmann exploration) which, during an iteration, associate with each action a a probability of being chosen as a function of rewards received at the previous iterations, when this action is selected in the state s.
Coming back to the example where both classifiers make use of an optimised look up table via the reinforcement learning, the implementation of the exploration/exploitation compromise is illustrated by
The time classifier CT then successively provides the different data flow categories to the frequency classifier as a function of the selected hierarchisation rule (i.e. depending on the selected action aCT or aCT*). An iterative process then starts, which is completed when all the data flow categories have been served or when the entirety of the resources has been allocated.
If depending on the selected action aCT or aCT*, the category p is to be scheduled at the nth iteration of this process, the state sp of the users belonging to this category p is provided to a second reinforcement learning module 3 in charge of determining the optimum policy in terms of user scheduling. A second exploitation/exploration compromise module 4 will select, with a probability 1−ε, the action aCF=up determined by the second reinforcement learning module 3 based on the policy learnt at this stage of iterations and, with a probability ε, a random action up*. As for the first module 2, mechanisms other than the ε-greedy (such as the Boltzmann exploration) can be selected to optimise the compromise between exploration and exploitation.
Then, a reward rp(sp,{aCS,aCT},{up,up*}) is associated with the selected action up or up*. The set of the actions up or up* implemented for each of the P categories is noted aCF. When the iterative process is completed (when all the data flow categories have been served or when the entirety of the resources has been allocated), an overall reward R(s,{aCT,aCT*},aCF) which depends on the rewards received for each category is associated with the action aCT or aCT* implemented by the time classifier CT. This overall reward is used to update the reinforcement learning modules 1, 3. This updating typically makes use of the difference between the expected reward which is based on the passed experiment and the new reward. When this difference becomes low enough (for example lower than 10−3), the learning process can be stopped and the learned policy can be stored in the look up table LUT. The classifiers CT and CF can then simply look up this table to select the optimum hierarchisation and scheduling rules for a given current state of the telecommunication system. It will be noted that in a possible embodiment described after, the function relating the state to the action is approximated by an approximation function, and the coefficients of the approximation function are stored in the look up table instead of the optimum policy.
Different alternative embodiments will be described in the following.
Representation of the System State
In order to reduce the complexity of the solution provided by the invention and to speed-up the learning process, different alternative embodiments can be implemented, independently or jointly, to simplify the representation of the system state. As previously mentioned, this state is comprised of controllable elements and uncontrollable elements. These alternative embodiments specifically aim at efficiently representing the state as a function of this classification.
Compression of the Controllable State Space
According to this alternative, the variables of the controllable state sp (for example delay, rate and packet loss rates) of each of the data flow categories are statistically treated, for example to calculate the average and variance thereof. In this manner, the controllable state is no longer dependent on the number of users Ip in each of the data flow categories. The dimension of the state is thus reduced from Σp∈T
Compression of the Uncontrollable State Space
According to this alternative, the variables of the uncontrollable state sp of each of the categories are statistically treated so as to reduce the size of this state. For some parameters, as for the CQI indicator, it is possible to remove the dependence on the number of users Ip in the category and to the bandwidth (i.e. the numbers NRB of available resource units). On the contrary, for other parameters, as the characteristics of the data flow, it is only possible to remove the dependency on the number of users. Clustering or classification algorithms can be used to that end. Among the clustering algorithms, Lloyd or Swap heuristics can be mentioned. Among the classification algorithms, support vector machines (SVM) or radial basis function based neural networks (RBFNN) can be mentioned.
Reward
An example of calculating a reward function has been presented in what precedes. However, other functions can be calculated, all being dependent on the KPI indicators and QoS requirements. The reward assigned to the time classifier can for example be normalised by the number of data flow categories, whereas the reward assigned to the frequency classifier can for example be normalised by the number of users and the number of KPI indicators in the category being considered.
Reinforcement Learning Techniques
Many reinforcement learning techniques have been proposed in the literature and can be used to create the look up table. Generally, the object of a reinforcement learning is to identify the policy π which minimises (respectively maximises) the expected value of a certain function J(π) representative of a cost (respectively a reward) associated with achieving the policy π.
This function J(π) can represent a sum of devaluated rewards:
where s0 is the initial state, rt+1 is the reward received when the policy π is implemented starting from state s0 and γ∈[0; 1] is a devaluation rate which determines the current value of future rewards.
The function J(π) can also be represented as an average reward according to:
Further, the reinforcement learning algorithms can be divided into three groups: pure critic, pure actor, actor-critic.
“Pure Critic” Methods
These methods learn the optimum strategy π by using a function Qπ(s,a) which evaluates the value of each state-action pair (s,a) whereas there is no explicit function to represent and evaluate the currently learnt policy. When the object is to maximise a devaluated reward sum, Qπ(s,a) can be defined according to:
According to the Bellman optimality equation, the optimum value Q*(s,a) can be calculated recursively as follows:
where s′ is the next state observed when the action a has been applied in the presence of the state s. After convergence, it is possible to determine the optimal action a* for the current state s, such that Q*(s,a) is maximum:
a*=argmaxa′∈AQ*(s,a).
Finally, the optimum state-action pairs are stored in the look up table.
“Pure Actor” Methods
These methods operating with a family of parametric polices aim at optimising the function J(π) directly on the parameter space of the policy. A performance gradient, with respect to the actor parameters, is directly estimated by simulation, and the parameters are updated in an improvement direction. The “pure actor” methods have the advantage with respect to “pure critic” methods of allowing the policy to generate actions in a completely continuous action space.
Let us consider a parameterisation of the policy π by the parameter vector θ. J(π) is a function of the parametric policy πθ, and is a function of θ. The J gradient with respect to θ is described as:
Then, by using conventional optimisation techniques (such as the gradient descent), a locally optimum solution of the function J can be identified. A drawback of the “pure actor” approach is that the estimated gradient can have a significant variance, which increases the duration required for learning.
“Actor-Critic” Methods
These methods aim at combining advantages from “pure actor” and “pure critic” methods. As for the “pure actor” methods, these methods are capable of prescribing continuous actions, whereas the significant variance of the “pure actor” methods is counterbalanced by adding a critic the role of which is to evaluate the current policy prescribed by the actor.
Function Approximating the Reinforcement Learning
The state-action space can turn out to be combinatory and huge. The problem is thereby not only the amount of memory necessary for the look up table, but also the time and data necessary to perform the optimum policy learning.
Approximation by a Linear Function
It is possible to resort to an approximating function Ĵ(·,θ) in the form of a linear function of a weight vector θ. Thereby, a characteristic vector φ(s)(φ1(s), . . . , φn(s))T having the same number of components as θ at corresponds to each state s. There are many manners for elaborating characteristics from the states, for example the Fourier base function or tile coding. Generally, the approximating function is given by the scalar product between θ and φ(s): Ĵ(·,θ)=θT·φ(s)=Σi=1nθi·φi(s).
The individual functions φi are called base functions because they form a linear base for all the linear functions of this form. Constructing a characteristic vector with n dimensions to represent the states is equivalent to selecting a set of n base functions. It is natural to use a stochastic gradient descent with a linear function approximation to iteratively update θ until convergence.
Approximation by Means of a Fuzzy Interference System
To efficiently represent the state of a system, a well-known solution consists in integrating a fuzzy inference system (FIS) in the Q-learning algorithm. The FIS system is based on the fuzzy logic in which, unlike the standard logic, the truth of any affirmation is a matter of degree, i.e. is noted by a value between 0 and 1. On the other hand, the FIS system uses linguistic variables, namely variables the values of which are words, as “high” or “low”. Each linguistic variable xi is associated with a set of terms T(xi) which includes all the fuzzy sets corresponding to the linguistic values of xi. The learning phase comprises three phases: fuzzification, truth value calculation and defuzzification.
Approximation by a Neural Network
The neural networks are a powerful tool to estimate the optimum function Ĵ(·,θ). As for the linear approximation approach, during the learning phase, at iteration i, the new weights θi are calculated such that the root mean square error in the Bellman equation is reduced. However, in this case, the approximating function is calculated as a non-linear function of the weights θ. More particularly, if a three-level predictive neural network is considered, with a hidden level comprised of D nodes and an input level comprised of M nodes, the output k can be calculated as:
where x corresponds to the inputs of the neural network, θ is the set of weights in the neural network, h is the activation function of the hidden nodes (a differentiable non-linear sigmoid function such as the logistic sigmoid or tangent hyperboloid), and σ is the activation function (the identity function is used here) of the output nodes.
The invention is thus concerned with a method for determining a resource allocation which implements the previously described reinforcement learning to determine the optimum resource allocation plan for a given state of the telecommunication system. This method can be implemented by a computer program product. The invention is also concerned with the exploitation of this learning and thus relates to a packet scheduling method comprising at each transmission time interval, selecting a resource allocation plan and a transmission resource allocation to the data flows in accordance with the selected resource allocation plan. This selection is made by querying a look up table the content of which results from the implementation of the reinforcement learning and which makes it possible to identify, from the current state of the multi-access telecommunication system, the resource allocation plan to be selected. The invention extends to a computer program product comprising program code instructions for executing the scheduling method when said program is executed by a computer, as well as a node of a multi-access telecommunication system which comprises a look up table the content of which results from the implementation of the reinforcement learning and a planner configured to, at each transmission interval, query the look up table to identify, from the current state of the multi-access telecommunication system, a resource allocation plan, and make a transmission resource allocation to the data flows in accordance with the identified resource allocation plan.
| Number | Date | Country | Kind |
|---|---|---|---|
| 17 59982 | Oct 2017 | FR | national |
