METHOD OF OPTIMIZING DEPENDENT TASK OFFLOADING IN INTERNET OF VEHICLES USING DEEP REINFORCEMENT LEARNING

Abstract

A method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning is provided, including the following steps: S1: constructing a vehicle system network; S2: constructing a task model for an application program; S3: constructing a task load model: calculating delay, energy consumption and incentive compensation for local computing, offloading to nearby vehicles and offloading to nearby RSUs three offloading methods based on the vehicle network system and the task model, respectively; S4: determining task priorities: first determining the priorities of each of subtasks according to allocation of predecessor nodes of the subtasks combining with dynamic network environment, and then scheduling according to a multi-queue algorithm; and S5: finding an optimal offloading strategy by using the deep reinforcement learning.

Description

TECHNICAL FIELD

The present disclosure relates to vehicle edge computing, and specifically relates to a method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning.

BACKGROUND

With the dramatic growth in the number of vehicle users worldwide and the increasing demand for services from vehicle users, telematics networks have endured more business challenges such as high transmission bandwidth requirements of the network, high transmission reliability and strong data processing capabilities. Internet of vehicles (IoV) integrates traditional vehicle Ad Hoc Network and telematics, and thus can effectively improve vehicle service capability. In IOV, intelligent vehicles can execute a variety of applications, such as collision warning, autonomous driving, and auto-navigation. However, these applications require a large amount of computational and storage resources and have strong delay requirements. It is a challenging task to execute these applications on resource-limited vehicles under low delay constraints.

In-vehicle edge computing that integrates Mobile Edge Computing (MEC) into IoV is a promising solution to effectively address the above problems. In-vehicle edge computing may improve vehicle quality of service by deploying computation of MEC servers and storage resources in the vicinity of vehicles. Compute-intensive and delay-sensitive tasks may be loaded onto the MEC servers for execution over a wireless network. Although in-vehicle edge computing may reduce the delay of task execution, the edge servers have limited computation and storage capacity and thus cannot guarantee load balancing. Therefore, an effective offloading strategy may reduce the processing delay of tasks and improve the quality of service for users.

Current in-vehicle edge computing mainly study the computational optimization of individual tasks and lack consideration of the dependencies between tasks, and thus limits the application scope and effectiveness of in-vehicle edge computing technologies. In addition, considering the environmental protection factors, most vehicles are electric vehicles, and energy consumption is crucial for the use of electric vehicles. In practical applications, vehicles or infrastructures providing computing services usually require corresponding incentive compensation, but existing studies rarely focus on energy consumption in computing and corresponding incentive compensation. To address these issues, the present disclosure aims to minimize the energy consumption and the incentive compensation paid during computation while satisfying the demand for computation services of an application, and thereby improve the quality of service for the user. Specifically, the present invention proposes a system and method for optimizing the offloading of dependent tasks in the IOV using deep reinforcement learning, taking into account the dependency relationships between the tasks so as to ensure that the delay constraints are satisfied while minimizing the energy consumption in the computation process and the incentive compensation paid to the vehicle or infrastructure for the computation service. Specifically, the computational resources, energy costs and incentive compensation of vehicles and infrastructures are first mathematically modeled, and then corresponding task scheduling and offloading strategies are developed using deep reinforcement learning techniques and incorporating the dependencies between tasks. This ensures that the delay constraints are satisfied while minimizing the energy consumption in the computation process and the incentive compensation paid to the vehicles or infrastructures providing the computation services.

In summary, the present disclosure proposes a novel in-vehicle edge computing technology that may better solve the computational offloading problem of dependencies between tasks, energy costs and incentive compensation, and improve the efficiency and quality of computing services.

SUMMARY

An objective of the present disclosure to provide a method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning, so as to overcome the shortcomings of the prior art.

In order to achieve the above objective, the present disclosure provides a method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning, comprising the following steps:

• • S1: constructing a vehicle system network, wherein the vehicle system network comprises MEC servers, RSUs and vehicular user sides connected in sequence;
  • S2: constructing a task model for an application program, wherein tasks with temporal dependencies and data dependencies are modeled as a DAG task model;
  • S3: constructing a task load model: calculating delay, energy consumption and incentive compensation for local computing, offloading to nearby vehicles and offloading to nearby RSUs three offloading methods based on the vehicle network system and the task model, respectively;
  • S4: determining task priorities: first determining the priorities of each of subtasks according to allocation of predecessor nodes of the subtasks combining with dynamic network environment, and then scheduling according to a multi-queue algorithm; and
  • S5: finding an optimal offloading strategy by using the deep reinforcement learning: each of the subtasks selects a corresponding execution device to complete the task offloading, so as to minimize energy consumption and paid incentive compensation under constraints of delay, and thereby improve quality of service for users.

Compared to the prior art, the present disclosure has the following technical effects.

This disclosure is applicable to V2I and V2V collaborative offloading scenarios in an IoV environment. The method decouples the nonlinear integer programming problem into two subproblems: the associated subtask scheduling problem and the task offloading problem. Combining the allocation of predecessor tasks of the subtasks and the dynamic network environment, the method prioritizes the set of ready subtasks and then solves the offloading problem using deep reinforcement learning based on value functions and policy functions. The method optimizes the task offloading problem, and minimizes the energy consumption during computation and the incentive compensation paid to the vehicles or infrastructures with the computation service, while ensuring task dependency and the constraints of delay. The present disclosure does not require excessive a priori knowledge, has good reusability in similar application scenarios, and thus is of high practical value.

The present disclosure provides a method for dependent task offloading in V2I and V2V collaborative edge computing based on deep reinforcement learning, effectively minimizes the weighted sum of the energy consumption and the incentive compensation paid to the client-side vehicle under the constraints of delay, and thereby improve the user experience. The process of the present disclosure is standardized, and is easy to operate. Moreover, the present disclosure also has strong practical value and reusability, and has a wide range of application prospects in similar application scenarios.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of the V2I and V2V collaborative offloading model of the embodiment in this disclosure.

FIG. 2 shows the schematic diagram of the deep reinforcement learning algorithm of the embodiment in this disclosure.

FIG. 3 is a flowchart based on deep reinforcement learning algorithm in an IoV scenario of the embodiment in this disclosure.

DESCRIPTION OF EMBODIMENTS

The present disclosure is further described below in connection with the drawings and specific embodiments so that those skilled in the art may better understand the present disclosure and be able to allow implementation, but the cited implementations are not intended to be a limitation of the present disclosure.

An agent takes an action in the current state to obtain a reward from the environment, and then enters the next state. Through continuous interaction, the agent may gradually learn the optimal offloading strategy. In this process, the agent constantly updates its value function to better estimate the value of each action. At the same time, the agent also continuously adjusts strategy in order to better explore the search space and find the optimal offloading decision.

In order to minimize the weighted sum of energy consumption and incentive compensation while satisfying delay constraints in an IoV edge computing scenario combining Vehicle-To-Infrastructure (V2I) offloading and Vehicle-To-Vehicle (V2V) offloading, the present disclosure provides a method for dependent task offloading based on deep reinforcement learning.

The edge computing scenario described in the present disclosure refers to vehicle nodes in an IoV network communication system combining V2I offloading and V2V offloading. These vehicle nodes may carry out V2I communication with edge Mobile Edge Computing (MEC) servers and may carry out V2V communication with other vehicle nodes. In this scenario, the vehicle that generates the application program is called a client-side vehicle and the vehicles that provide computational services to the application program are called server-side vehicles. The application program includes a series of indivisible subtasks with temporal dependencies and data dependencies that can be represented using a directed acyclic graph (DAG). These subtasks can be executed locally, transferred to server-side vehicles for execution, or uploaded to Rode side Units (RSUs) next to the highway for execution. In order to achieve minimizing the weighted sum of energy consumption and incentive compensation in this scenario, this disclosure provides a method for dependent task offloading based on deep reinforcement learning, which requires very little a priori knowledge and simple interactions with the environment to gain learning experience and obtains a near-optimal solution for minimizing the weighted sum of energy consumption and incentive compensation.

As shown in FIG. 3, the present disclosure provides a method of optimizing dependent task offloading in an IoV using deep reinforcement learning, including the following steps.

S1: Constructing a vehicle system network.

The proposed vehicle system network includes MEC servers, RSUs and vehicular user sides connected in sequence. A plurality of vehicle user sides is connected to each other via V2V transmission links to realize inter-vehicle communication and collaborative computing. Vehicle user sides and RSUs are connected via V2I transmission links to realize the communication between vehicles and the network. The MEC servers, as edge computing resource providers providing resources such as computation, storage and network, may provide edge computing services for the vehicle user sides. RSUs, as the infrastructures of the network, provide information such as network connection and road conditions and provide data and collaborative computation support for the vehicle user sides. The whole vehicle system network constitutes a distributed edge computing platform, and realizes cooperative computing and communication between vehicles and between vehicles and networks.

As shown in FIG. 1, in this scenario, according to roles in task processing, vehicles can be divided into a client-side vehicle and server-side vehicles. The data transmission links include V2V transmission links for task data, the upload links for task data to RSUs, the connection links between RSUs and servers, and the connection links between RSUs and RSUs. For the client-side vehicle, its tasks can be processed by the following three offloading methods: 1) Local computing: the client-side vehicle processes tasks by itself; 2) Offloading to nearby vehicles: the client-side vehicle uploads the tasks to neighboring server-side vehicles; 3) Offloading to nearby RSUs: the client-side vehicle uploads tasks to nearby RSUs.

S2: Constructing a task model for the application program:

Tasks with temporal dependencies and data dependencies are modeled as a DAG task model, and the DAG task is modeled as G=(V, E), where the vertex set V={ϕ_i|1≤i≤N}, N denotes number of subtasks in V, the triad is defined as ϕ_i=(w_i, d_i, τ_i), w_i, d_i, τ_i denote the computational workload, the data size, and the maximum completion time tolerance delay of the task ϕ_i respectively, E denotes the set of edges, and each edge e_ϕi,ϕj∈E is a directed edge denoting the priority of the execution between them, indicating that the subtask ϕ_i needs to be completed before ϕ_j (which is equivalent to the fact that the output of ϕ_i is used as a part of the input of ϕ_j). If an edge exists from ϕ_i to ϕ_j, ϕ_i is call as an immediate predecessor of ϕ_j; otherwise, ϕ_j is called an immediate successor of ϕ_i. All predecessors need to be completed before their successors. A subtask without any predecessor node is a start node, also known as a source task, denoted as ϕ_entry; a subtask without any successor node is an exit node, denoted as ϕ_exit. With the DAG task model, the temporal dependencies and data dependencies between tasks can be clearly represented, which provides a basis for task scheduling.

S3: Constructing the task load model: based on the vehicle network system and the task model, calculating the delay, energy consumption and incentive compensation for the three offloading methods, namely, local computing, offloading to nearby vehicles, and offloading to nearby RSUs, respectively

S301: Interdependent subtasks executed at different vehicle nodes require data transfer. Specifically, it takes some time to transfer the computation results of subtask ϕ_j executed on vehicle node m to subtask ϕ_i executed on vehicle node n, and this data transfer time is defined as follows:

$\begin{array}{cc}{\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}\left(m,n\right)=\{\begin{array}{cc}\frac{e_{{\varphi }_j{\varphi }_i}}{r_{\mathrm{mn}}},& e_{\mathrm{mn}}\in E^{{\mathrm{RSU}}_i}\\ \min \left(\frac{e_{{\varphi }_j{\varphi }_i}}{r_{\mathrm{mk}}}+\frac{e_{{\varphi }_j{\varphi }_i}}{r_{\mathrm{kn}}}\right),& e_{\mathrm{mk}}\in E^{{\mathrm{RSU}}_1}\mathrm{and}{e}_{\mathrm{kn}}\in E^{{\mathrm{RSU}}_1}\mathrm{and}{e}_{\mathrm{mn}}\notin E^{{\mathrm{RSU}}_1}\end{array},& \left(1\right)\end{array}$

• • where k, m and n represent the labelings of the vehicle nodes respectively ϕ_j is the predecessor node of subtask of ϕ_i. r_mn represents the transmission bandwidth between the vehicle node m to the vehicle node n e_ϕjϕi denotes the data size of the dependent data of the subtasks ϕ_i to ϕ_j, e_mn denotes whether or not the vehicle nodes m and n are within the communication range, and E^RSUi denotes the set of edges established by vehicles through single-hop V2V communication within the communication range of road side unit RSU_i.

S302: Calculating the time to execute the task locally.

Specifically, for the case of executing the task locally, the completion time of the subtask ϕ_i includes the waiting time t_i^local,wait and the compution time t_i^{local,compution}.

$\begin{array}{cc}{t}_i^{\mathrm{local},\mathrm{complete}}=t_i^{\mathrm{local},\mathrm{wait}}+t_i^{\mathrm{local},\mathrm{compution}},& \left(2\right)\end{array}$ $\begin{array}{cc}{t}_i^{\mathrm{local},\mathrm{compution}}=\frac{w_i}{f_c},& \left(3\right)\end{array}$ $\begin{array}{cc}{t}_i^{\mathrm{local},\mathrm{wait}}=\max \left\{\mathrm{avail}\left(V_c\right),\max \left({\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}\left(m,V_c\right)\right)\right\},& \left(4\right)\end{array}$

Where V_c denotes the client-side vehicle, avail(V_c) denotes the time when the client-side vehicle finishes all the scheduled tasks, w_i denotes the amount of computation required by the subtask ϕ_i, and f_c denotes the computational power of the client-side vehicle.

For the case of executing tasks on RSU_j, the completion time of subtask ϕ_i includes upload time t_i^RSUj,upload, waiting time t_i^RSUj,wait and compution time t_i^{RSUj,compution}.

$\begin{array}{cc}{t}_i^{{\mathrm{RSU}}_j,\mathrm{complete}}=t_i^{{\mathrm{RSU}}_j,\mathrm{upload}}+t_i^{{\mathrm{RSU}}_j,\mathrm{wait}}+t_i^{{\mathrm{RSU}}_j,\mathrm{compution}},& \left(5\right)\end{array}$ $\begin{array}{cc}{t}_i^{{\mathrm{RSU}}_j,\mathrm{upload}}=\frac{d_i}{r_{V_c,{\mathrm{RSU}}_j}},& \left(6\right)\end{array}$ $\begin{array}{cc}{t}_i^{{\mathrm{RSU}}_j,\mathrm{wait}}=\max \left\{\mathrm{avail}\left({\mathrm{RSU}}_j\right),\max \left({\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}\left(m,{\mathrm{RSU}}_j\right)\right)\right\},& \left(7\right)\end{array}$ $\begin{array}{cc}{t}_i^{{\mathrm{RSU}}_j,\mathrm{compution}}=\frac{w_i}{f_{{\mathrm{RSU}}_j}},& \left(8\right)\end{array}$

Where r_Vc,RSUj denotes the transmission rate between the client-side vehicle and the specified RSU_j, d_i denotes the data size of the subtask ϕ_i, and f_RSUj denotes the computational power of the road side unit RSU_j.

Similarly, the time to execute task on the server-side vehicles includes transmission time t_i^Vk,trans, waiting time t_i^Vk,wait and compution time t_i^Vk,compution.

$\begin{array}{cc}{t}_i^{V_k,\mathrm{complete}}=t_i^{V_k,\mathrm{trans}}+t_i^{V_k,\mathrm{wait}}+t_i^{V_k,\mathrm{compution}},& \left(9\right)\end{array}$ $\begin{array}{cc}{t}_i^{V_k,\mathrm{trans}}=\frac{d_i}{r_{V_c,V_k}},& \left(10\right)\end{array}$ $\begin{array}{cc}{t}_i^{V_k,\mathrm{wait}}=\max \left\{\mathrm{avail}\left(V_k\right),\max \left({\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}\left(m,V_k\right)\right)\right\},& \left(11\right)\end{array}$ $\begin{array}{cc}{t}_i^{V_k,\mathrm{compution}}=\frac{w_i}{f_k},& \left(12\right)\end{array}$

Where r_Vc,Vk denotes the transmission rate between the client-side vehicle V_c to the specified server-side vehicle V_k, and f_k denotes the computational power of the server-side vehicle V_k.

S303: Calculating the energy consumption for executing tasks locally:

$\begin{array}{cc}{e}_i^{\mathrm{local}}={\xi \left(t_i^{\mathrm{local},\mathrm{compution}}\right)}^{-2}{\left(w_i\right)}^3,& \left(13\right)\end{array}$

Where ξ is a coefficient related to the architecture of the computer chip, and there is no incentive compensation because it is executed locally;

• • calculating the energy consumption for executing the task at the RSUs:

$\begin{array}{cc}{e}_i^{{\mathrm{RSU}}_j}=p_{V_c}{t}_i^{{\mathrm{RSU}}_j,\mathrm{upload}},& \left(14\right)\end{array}$

• • where p denotes the upload power, so the incentive compensation to be paid to the road side RSU_j unit for processing subtask ϕ_i is as follows:

$\begin{array}{cc}{C}_{{\mathrm{RSU}}_j,{\varphi }_i}=p_j{t}_i^{{\mathrm{RSU}}_j,\mathrm{compution}},& \left(15\right)\end{array}$

• • where p_j denotes the fee charged by the road side unit RSU_j per unit time (s);
  • calculating the energy consumption for executing the task on the server-side vehicles:

$\begin{array}{cc}{e}_i^{V_k}=p_{V_c}{t}_i^{V_k,\mathrm{trans}},& \left(16\right)\end{array}$

• • where the incentive compensation to be paid to the server-side vehicle k for processing the subtask ϕ_i is as follows:

C _{V k ,ϕ i} =w _iρ_k  (17),

• • where ρ_k denotes the fee charged by the server-side vehicle k per unit of resource (1G clocks).

S304: The time t_i, the energy consumption e_i and the cost c_i to be paid for executing a subtask ϕ_i are:

$\begin{array}{cc}{t}_i=x_i^{\mathrm{local}}{t}_i^{\mathrm{local},\mathrm{complete}}+x_i^{{\mathrm{RSU}}_j}{t}_i^{{\mathrm{RSU}}_j,\mathrm{complete}}+x_i^{V_k}{t}_i^{V_k,\mathrm{complete}},& \left(18\right)\end{array}$ $\begin{array}{cc}{e}_i=x_i^{\mathrm{local}}{e}_i^{\mathrm{local}}+x_i^{{\mathrm{RSU}}_j}{e}_i^{{\mathrm{RSU}}_j}+x_i^{V_k}{e}_i^{V_k},& \left(19\right)\end{array}$ $\begin{array}{cc}{c}_i=x_i^{{\mathrm{RSU}}_j}{C}_{{\mathrm{RSU}}_j,{\varphi }_i}+x_i^{V_k}{C}_{V_k,{\varphi }_i},& \left(20\right)\end{array}$

• • where x_i^local, x_i^RSUj and x_i^Vk denote the offloading decisions of the subtasks, and the values of x_i^local, x_i^RSUj and x_i^Vk take all 0 or 1, and only one of the three can take 1.

Total incentive compensation cost, i.e., the incentive compensation to be paid to the client-side vehicle to complete all tasks, is denoted as:

$\begin{array}{cc}C\left(X\right)={\sum }_{i=1}^N{c}_i,& \left(21\right)\end{array}$

Total energy consumption cost, i.e., the energy consumption of the client-side vehicle to complete all tasks, denoted as:

$\begin{array}{cc}E\left(X\right)={\sum }_{i=1}^N{e}_i,& \left(22\right)\end{array}$

Where X denotes the offloading decision vector and N denotes the number of tasks.

These formulas describe the relationship between task offloading decisions, execution time, energy consumption, and incentive compensation, and can be used to optimize the task offloading scheme to minimize the total energy consumption and total incentive compensation.

S305: In order to make the optimal offloading decision, an optimization problem considering the energy consumption and incentive compensation for completing the tasks is created:

$\begin{array}{cc}\begin{array}{c}𝒫:\underset{x}{\min }\phi \left(X\right)=\alpha C\left(X\right)+\beta E\left(X\right)\\ s.t.C1:x_i^{\mathrm{local}}+x_i^{{\mathrm{RSU}}_j}+x_i^{V_k}=1,\\ x_i^{\mathrm{local}}\in \left\{0,1\right\}{x}_i^{{\mathrm{RSU}}_j}\in \left\{0,1\right\}{x}_i^{V_k}\in \left\{0,1\right\},\\ C2:t_i\le \min \left(l_{c,k},{\tau }_i\right),x_i^{V_k}=1\\ C3:t_i\le \min \left(l_{c,{\mathrm{RSU}}_j},{\tau }_i\right),x_i^{{\mathrm{RSU}}_j}=1\\ C4:\alpha +\beta =1,\alpha >0\beta >0\end{array}& \left(23\right)\end{array}$

• • where C(X) denotes the total incentive compensation cost, E(X) denotes the total computational cost, C1 denotes that each subtask has to be offloaded to a certain device, C2 and C3 denote that if a subtask is offloaded to server-side vehicles or RSUs, completion time cannot exceed the maximum allowable time thresholds l_c,k or l_c,RSUj, and C4 denotes that the weighted sum of energy consumption and incentive compensation is 1, and both must be positive number.

The optimization problem may be decomposed into two subproblems: task priority determination and offloading decision. First, the task priority determination is to determine the execution order of each subtask in order to satisfy the constraints of delay and task dependency. Then, the task scheduling order is used for offloading decisions, the deep reinforcement learning is used to decide whether each subtask is executed locally, offloaded to the RSUs, or offloaded to the server-side vehicles to minimize the weighted sum of energy consumption and incentive compensation.

S4: Determining task priority: first determining the priority of each subtask according to the allocation of the predecessor node of the subtask combining with the dynamic network environment, and then scheduling according to the multi-queue algorithm.

S401: Starting subtasks, assuming that Rank^D(ϕ_ready)=0, determining the priority of the successor subtasks based on the scheduling arrangement of the predecessor subtasks, determining the ranking of the existing subtasks ϕ_i∈ϕ_ready, and computing the value of the dynamic down-ranking as follows:

$\begin{array}{cc}{\mathrm{Rank}}^D\left({\varphi }_i\right)=\underset{{\varphi }_j\in \mathrm{pred}\left({\varphi }_i\right)}{\max }\left\{{\mathrm{Rank}}^D\left({\varphi }_j\right)+\stackrel{\_}{{\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}^D}\right\}+\stackrel{\_}{{\mathrm{CT}}_{{\varphi }_i}^D}.& \left(24\right)\end{array}$

• • where pred(ϕ_i) is the set of immediate predecessor nodes of subtask ϕ_i, CT_ϕi^D is the dynamic average compution time, which may be affected by the allocation of ϕ_j∈pred(ϕ_i), C^VC(ϕ_i) denotes the set of candidate server-side vehicles or infrastructures that may execute subtask ϕ_i, and ϕ_ready denotes the set of ready subtasks and is defined as follows:

$\begin{array}{cc}{\stackrel{\_}{{\mathrm{CT}}_{{\varphi }_i}}}^D=\frac{{\sum }_{m\in C^{{\mathrm{VC}}_{\left({\varphi }_i\right)}}}\mathrm{CT}\left({\varphi }_i,m\right)}{❘C^{\mathrm{VC}}\left({\varphi }_i\right)❘}{\varphi }_i\in {\varphi }_{\mathrm{ready}},& \left(25\right)\end{array}$

• • moreover, TT_ϕj,ϕi^D represents the dynamic average transmission time, which is also affected by the allocation of ϕ_j∈pred(ϕ_i), and is defined as follows:

$\begin{array}{cc}{\stackrel{\_}{{\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}}}^D=\frac{{\sum }_{m\in C^{{\mathrm{VC}}_{\left({\varphi }_i\right)}}}{\mathrm{TT}}_{{\varphi }_j,{\varphi }_i}\left(n,m\right)}{❘C^{\mathrm{VC}}\left({\varphi }_i\right)❘}{\varphi }_i\in {\varphi }_{\mathrm{ready}}{\varphi }_j\in \mathrm{pred}\left({\varphi }_i\right).& \left(26\right)\end{array}$

S402: Task scheduling depends not only on priority but also on dependencies between tasks. A subtask can only be executed when all its predecessor subtasks are completed. Therefore, it is necessary to maintain a queue of ready subtasks, Q^ready, and a queue of executing subtasks, Q^exe, along with a queue of computed subtasks, Q^finished, and a set of all subtasks in the application program, T^task.

Initially, ϕ_entry subtasks are placed into Q^ready from the set T^task, the tasks in Q^ready are sorted in ascending order according to the subtask priority determination formula, the queue head element is removed to place the tasks into the queue Q^exe, of executing subtasks, and if there are completed tasks in queue Q^exe, the completed tasks are placed into the Q^finished queue. This may generate new ready subtasks, and if all predecessor node subtasks of a subtask ϕ_i in the set T^task are in the queue Q^finished, the subtask ϕ_i becomes a ready subtask, which is placed in the queue Q^ready, and the above steps are repeated until the set T^task is empty. In the whole process of subtask priority determination, the ready subtasks are dynamically determined according to the execution results of the subtask predecessor, and thus the priority of the ready subtasks is determined.

S5: Finding the optimal offloading strategy by using deep reinforcement learning, each subtask selects the corresponding execution device to complete the task offloading, so as to minimize energy consumption and incentive compensation under constraints of delay, thereby improving the quality of service for users, as shown in FIG. 2.

In deep reinforcement learning methods, the design of actions, states, and reward values plays a crucial role. In order to adapt to the system environment and enable the agent to output the optimal action based on the current environmental state. The state space, action space, and reward function are designed as follows:

State space S: the state is represented in vector form, indicating the state of the client-side vehicle, including task information, location, available computing resources, and connection time. State space S_t={T, N}, where T and N respectively represent the set of candidate vehicles and the set of ready subtasks available within the communication range of the road side unit RSU_j,

$\begin{array}{cc}T=\left(V^{{\mathrm{RSU}}_j}\right),N=\left(w_i,{\tau }_i,\mathrm{succ}\left({\varphi }_i\right),\mathrm{pred}\left({\varphi }_i\right)\right),& \left(27\right)\end{array}$

Action space A: the action is a decision made by the requester through the offloading strategy π: A←S, and has a direct impact on the external environment. The action at time t is denoted as A_t={x_i^local, x_i^RSUj, x_i^Vk}, x_i{circumflex over ( )}(V_k)}, which is a one-hot encoding. The value of each element can be 0 or 1.

Reward function R: intuitively speaking, the Reward R←(S, A), the feedback obtained by the agent during the offloading process, directly determines the strategy. The optimization goal is to minimize energy consumption and overhead costs, but in reality, each decision can only serve one subtask. Therefore, the cost of one subtask m at time t is represented as R_t(s, a)=−(e_i+c_i).

$\begin{array}{cc}{R}_t\left(s,a\right)=\{\begin{array}{cc}-e_i^{\mathrm{local}},& x_i^{\mathrm{local}}=1\\ -\left(e_i^{{\mathrm{RSU}}_j}+C_{\mathrm{RSU},j,{\Phi }_i}\right),& x_i^{{\mathrm{RSU}}_j}=1\\ -\left(e_i^{V_k}+C_{V_k,{\Phi }_i}\right),& x_i^{V_k}=1\end{array},& \left(28\right)\end{array}$

S501: Initializing the experience replay buffer and defining the maximum number of executions and initial episodes for training and evaluation.

S502: After initialization, initializing the parameters μ and θ of the current actor and current critic. This is because before starting training, the network parameters need to be initialized to some random values in order to avoid getting stuck in local optima during the learning process.

S503: Initializing the target actor's parameters μ′←μ and the target critic parameters θ′←θ separately. These networks are used during the algorithm training process to calculate TD errors and update the parameters of the current network.

S504: Obtaining the initial state s_t of the environment is an important step in the algorithm. At the same time, it is necessary to prepare the initial random noise n_t for action exploration. Through such exploration, more pairs of states and actions can be discovered, and the network can better converge to the optimal solution.

S505: From time t=1 until t=n (where n represents the number of indivisible subtasks in the application program). a_t is obtained based on the output of the Actor-C network of the current network and the random noise n_t:

$\begin{array}{cc}{a}_t={\phi }_{\mu }\left(s_t\right)+n_t,& \left(29\right)\end{array}$

S506: Executing action a_t, receiving reward r_t, and the environment state changing to s_t+1.

S507: Storing (s_t, a_t, r_t, s_t+1) in the experience replay buffer, utilizing the historical data in the experience replay buffer for offline training and improving the stability and robustness of the model.

S508: Sampling N tuples {(s_t, a_t, r_t, s_t+1)} a from the experience replay buffer for training the target network and updating the current network, where usually, random sampling is used to ensure the randomness of the sample.

S509: Using the target network to calculate the target value for each tuple y_t, where y_t is a value related to the current state s_t and action a_t, y_t includes the current reward r_t and the target predicted value of the next state s_(t+1). Specifically, the target value is calculated using the following formula:

$\begin{array}{cc}{y}_t=r_t+{\mathrm{\gamma \vartheta }}_{{\theta }^{\prime }}\left(s_{t+1},{\phi }_{{\mu }^{\prime }}\left(s_{t+1}\right)\right),& \left(30\right)\end{array}$

• • where γ is the discount factor, ϑ_θ′ is the target critic, φ_μ′ is the target actor, and these network parameters are initialized in S503.

S510: updating the parameter θ of the current critic by minimizing target loss L(θ), where L(θ) is calculated from the mean squared error (MSE) loss function as follows:

$\begin{array}{cc}L\left(\theta \right)=\frac{1}{N}{\sum }_{i=1}^N\left(y_i-{{\vartheta }_{\theta }\left(s_i,a_i❘\theta \right)}^2\right),& \left(31\right)\end{array}$

• • where N is the number of tuples sampled from the experience replay buffer, y_i is the target value, ϑ_θ(s_i, a_i|θ) is the output value of the current critic, and this MSE loss function is a common supervised learning loss function that can reflect the fitting ability of the current critic and help the network better adapt to the target value y_i.

S511: Updating the Actor of the current network by calculating the sampled policy gradient, where Q(s, a|θ) represents the predicted values of the current critic for state s and action a, ∇_aQ(s, a|θ) represents the action gradient on the predicted values, φ_u(s|μ) represents the action output of the current actor for state s, ∇_uφ_u(s|μ) represents the parameter gradient for the action output. The calculation formula for the gradient of this strategy is a Monte Carlo method that estimates the expected value using sampled tuple data.

$\begin{array}{cc}{\nabla }_{u}J\approx \frac{1}{N}{\sum }_{i=1}^N\left({\nabla }_{a}Q\left(s,a❘\theta \right){❘}_{s=s_i,a={\phi }_u\left(s_i\right)},{\nabla }_u{\phi }_u\left(s❘\mu \right){❘}_{s=s_i}\right),& \left(32\right)\end{array}$

S512: if the current iterative times t can be divided by C, i.e. t % C==0, updating the network parameters, where θ′ and u′ respectively represent the parameters of the target critic and the target actor, and v is a hyperparameter less than 1 used for smoothly updating:

$\begin{array}{cc}{\theta }^{\prime }=v\theta +\left(1-v\right){\theta }^{\prime },& \left(33\right)\end{array}$ $\begin{array}{cc}{u}^{\prime }=\mathrm{vu}+\left(1-v\right)u^{\prime },& \left(34\right)\end{array}$

S513: t=t+1, repeating S505 to S512.

S514: If the current iterative times t is equal to n, that is, the maximum execution count has been reached, then episode=episode+1, then repeating S504 to S513 until the training process is complete and the optimal offloading strategy is obtained.

It should be noted that the parts not elaborated in this description belong to the existing technology. Relevant technical personnel should understand that the above implementation is only intended to assist readers in understanding the principles and implementation methods of this disclosure, and the scope protected by this disclosure is not limited to such embodiments. Any equivalent substitution made on the basis of this disclosure is within the scope of protection of the rights of this disclosure.

Claims

1. A method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning, comprising the following steps: S1: constructing a vehicle system network, wherein the vehicle system network comprises MEC servers, RSUs and vehicular user sides connected in sequence;S2: constructing a task model for an application program, wherein tasks with temporal dependencies and data dependencies are modeled as a DAG task model;S3: constructing a task load model: calculating delay, energy consumption and incentive compensation for local computing, offloading to nearby vehicles and offloading to nearby RSUs three offloading methods based on the vehicle network system and the task model, respectively;S4: determining task priorities: first determining the priorities of each of subtasks according to allocation of predecessor nodes of the subtasks combining with dynamic network environment, and then scheduling according to a multi-queue algorithm; andS5: finding an optimal offloading strategy by using the deep reinforcement learning: each of the subtasks selects a corresponding execution device to complete the task offloading, so as to minimize energy consumption and paid incentive compensation under constraints of delay, and thereby improve quality of service for users.

2. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 1, wherein in S2, constructing the task model for the application program comprises as follows: the tasks with the temporal dependencies and the data dependencies are modeled as the DAG task model, and the DAG task is modeled as G=(V,E), wherein a vertex set V={ϕi|1≤i≤N}, N denotes a number of the subtasks in V; a triad is defined as ϕi=(wi, di, τi), wherein wi, di, τi denote a computational workload, a data size, and a maximum completion time tolerance delay of the tasks ϕi, respectively; E denotes a set of edges, and each of the edges eϕi,ϕj∈E is a directed edge denoting the priorities of execution between the edges, indicating that the subtasks ϕi need to be completed before ϕj, and output of ϕi is used as a part of input of ϕj; if an edge exists from ϕi to ϕj, ϕi is call as immediate predecessors of ϕj; otherwise, ϕj is called immediate successors of ϕi; all the predecessors need to be completed before successors; a subtask without any predecessor node is a start node, also known as a source task, denoted as ϕentry; a subtask without any successor node is an exit node, denoted as ϕexit; by the DAG task model, the temporal dependencies and data dependencies between the tasks are clearly represented, and a basis for task scheduling is provided.

3. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 2, wherein S3 comprises the following steps: S301: interdependent subtasks executed at different vehicle nodes require data transfer;S302: calculating time for executing the tasks locally;S303: calculating the energy consumption for executing the tasks locally;S304: time ti, energy consumption ei and cost ci to be paid for executing a subtask ϕi; andS305: creating an optimization problem considering the energy consumption and the incentive compensation for completing the tasks, so as to make an optimal offloading decision.

4. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 3, wherein S301 comprises as follows: there is some time to transfer computation results of the subtask ϕj executed on the vehicle node m to subtask ϕi executed on the vehicle node n, and the data transfer time is defined as follows:

5. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 3, wherein S302 comprises as follows: for a case of executing the tasks locally, a completion time of the subtask ϕi comprises a waiting time tilocal,wait and a compution time tilocal,compution,

6. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 5, wherein S303 comprises as follows:

7. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 6, wherein in S304, the time ti, the energy consumption ei and the cost ci to be paid for executing the subtask ϕi are:

8. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 7, wherein in S305, the optimization problem considering the energy consumption and the incentive compensation for completing the tasks is created:

9. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 8, wherein S4 comprises the followings steps: S401: starting the subtasks, assuming that RankD(ϕready)=0, determining the priority of the successor subtasks based on scheduling arrangement of the predecessor subtasks, determining a ranking of the existing subtasks ϕi∈ϕready, and computing a value of dynamic down-ranking as follows:

10. The method of optimizing dependent task offloading in an Internet of Vehicles using deep reinforcement learning according to claim 9, wherein S5 comprises the following steps: S501: initializing an experience replay buffer and defining a maximum number of the execution and initial episodes;S502: initializing parameters μ and θ of current actor and current critic after initializing the experience replay buffer;S503: initializing a parameter μ′←μ of target actor and a parameter θ′←θ of target critic separately;S504: obtaining an initial environment state st and preparing an initial random noise nt for action exploration;S505: obtaining an action at from time t=1 until t=n based on output of an Actor-C network of a current network and the random noise nt: