Patent Application
20230069604
In telecommunications networks, the Radio Access Network (RAN) performs more and more functions with each iteration of the telecommunications standards. That is, in order to enable the advantages of 5G over previous standards, the 5G RAN performs various additional functions. These RAN functions are situated between user devices and the core network, and are thus often performed at the base stations (e.g., cell towers) where computing power can be limited.
Some embodiments provide novel RAN intelligent controllers (RICs) for a telecommunication network. For instance, to provide a low latency near RT RIC, some embodiments separate the RIC functions into several different components that operate on different machines (e.g., execute on VMs or Pods) operating on the same host computer or different host computers. Some embodiments also provide high speed interfaces between these machines. Some or all of these interfaces operate in non-blocking, lockless manner in order to ensure that critical near RT RIC operations (e.g., datapath processes) are not delayed due to multiple requests causing one or more components to stall. In addition, each of these RIC components also has an internal architecture that is designed to operate in a non-blocking manner so that no one process of a component can block the operation of another process of the component. All of these low latency features allow the near RT RIC to serve as a high speed IO between the base station nodes (i.e., E2 nodes) and the control plane applications (e.g., xApps).
In some embodiments, the near RT RIC includes a datapath Pod, a service Pod, and an SDL (shared data layer) Pod. Part of the RIC’s low latency architecture is attributable to using different Pods to implement the data IO, service and SDL operations, so that different resource allocations and management operations can be provided to each of these Pods based on its respective needs of the operations that they perform. Also, in some embodiments, the RIC provides low-latency messaging between its various Pods.
The service Pod performs application (e.g., xApp) onboarding, registration, FCAPS (fault, configure, accounting, performance, security), and other services in some embodiments. It also provides services (such as metric collection, policy provisioning and configuration) to other RIC components. The SDL Pod implements the shared data layer of the near RT RIC. The SDL Pod in some embodiments also executes one or more service containers to execute one or more preprocessing or post-processing services on the data stored in the SDL.
The datapath Pod performs the data message forwarding between the base station components of the telecommunication network and control and edge applications of this network. In some embodiments, some or all of the datapath services of the datapath Pod are embedded in a datapath thread and a control thread of the datapath Pod. In other embodiments, the datapath services are embedded in a data IO thread, multiple data processing threads (DPTs) and a control thread.
The control thread in some embodiments is the interface with the service Pod and SDL Pod for the datapath threads, while in other embodiments it is the interface to just the service Pod for the datapath threads (as the datapath threads can communicate directly with the SDL Pod). The control thread in either of these approaches performs the slower, control related operations of the datapath, while the one or more datapath threads perform the faster IO operations of the datapath. The control thread in some embodiments interfaces with the service Pod to receive configuration data for configuring its own operations as well as the operations of the datapath thread.
The embodiments that separate the datapath thread into a data IO thread and multiple data processing threads further optimize the data IO by pushing the more computationally intensive operations of the datapath thread into multiple datapath processing threads, which then allows the less computationally intensive operations to run in the data IO thread. Both of these optimizations are meant to ensure a fast datapath IO (one that does not experience unwanted latencies) so that the near RT RIC can serve as a high speed interface between base station nodes and the control and edge applications.
Some embodiments use CRDs (custom resource definitions) to define resources in an O-RAN system. These resources include applications (xApps, rApps, etc.), components of these applications, deployment constructs, cloud constructs, AI/ML models and RTE constructs. Using these CRD descriptors, app developers can develop and distribute their applications in a platform independent manner.
The CRDs of some embodiments are used to on board applications into an O-RAN system. In some embodiments, the application on-boarding is secure, portable, developer friendly, follow more standard approaches (e.g., uses an open source approach) as opposed to custom solutions, allow RIC specific customization to the pods, and have an easy to interface with other tools.
To use CRDs, some embodiments use a new controller to process each CRD in the O-RAN system. In some embodiments, the SMO (Service, Management, Orchestration) framework of the O-RAN system has a service for processing CRDs. In some of these embodiments, the new SMO service has one controller to process the new application CRD. In some embodiments, the SMO identifies clouds for placements of the deployed applications and directs Kubernetes managers of the identified clouds to deploy the applications.
The use of CRDs has several advantages. They allow standardization with new schema that helps to inter-operate SMO and RIC applications. The CRDs provide control over the resource being created, network access etc. Lifecycle management can be owned by SMO. They also allow for policy enforcement at onboarding time. CRDs also enable faster application validation and certification processes. They make application more portable because the same CRD can be implemented in different O-RAN systems and by O-RAN components of different vendors. CRDs also hide implementation of the advanced app configuration (example: SRIOV, shared memory etc.).
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Today, there is a push to have the Radio Access Network (RAN) of a telecommunication network (e.g., a cellular network) implemented as O-RAN, a standard for allowing interoperability for RAN elements and interfaces.
As defined in the standard, the SMO 110 in some embodiments includes an integration fabric that allows the SMO to connect to and manage the RIC 115, the managed functions 120-130, and the O-Cloud 140 via the open interfaces 150. Unlike these elements, the O-RU 135 is not managed by the SMO 110, and is instead managed by the O-DU 130, as indicated by the dashed line 160, in some embodiments. In some embodiments, the O-RU 135 processes and sends radio frequencies to the O-DU 130.
In some embodiments, the managed functions 120-130 are logical nodes that each host a set of protocols. According to the O-RAN standard, for example, the O-CU-CP 120, in some embodiments, include protocols such as radio resource control (RRC) and the control plane portion of packet data convergence protocol (PDCP), while the O-CU-UP 125 includes protocols such as service data adaptation protocol (SDAP), and the user plane portion of packet data convergence protocol (PDCP).
The two RICs are each adapted to specific control loop and latency requirements. The near real-time RIC 115 provides programmatic control of open centralized units (O-CUs) and open distributed units (O-DUs) on time cycles of 10 ms to 1 second. The non-real-time RIC (non-RT RIC) 105, on the other hand, provides higher layer policies that can be implemented in the RAN either via the near-RT RIC or via a direct connection to RAN nodes. The non-RT RIC is used for control loops of more than 1 second. Each RIC 105 or 115 serves as a platform on which RAN control applications execute. These applications can be developed by third-party suppliers that are different from the RIC vendors. These applications are referred to as “xApps” (for the near-RT RIC 115) and “rApps” (for the non-RT RIC).
The near real-time RIC 115, in some embodiments, is a logical aggregation of several functions that use data collection and communications over the interfaces 155 in order to control the managed functions 120-130. In some embodiments, the non-real-time RIC 105 uses machine learning and model training in order to manage and optimize the managed functions 120-130. The near RT RIC in some of these embodiments also uses machine learning.
In some embodiments, the O-Cloud 140 is responsible for creating and hosting virtual network functions (VNFs) for use by the RIC 115 and the managed functions 120-130. In some embodiments, the DU is in charge of per-slot decisions of user scheduling and includes RAN scheduler that performs MAC control assistance and user-level tracing. In order to increase computing power available in the cloud (i.e., compared to base stations that typically execute the RAN functions), the RIC is implemented in one or more public and/or private cloud datacenters and implements an improved cloudified RAN scheduler in the cloud, thereby offloading these MAC control assistance and user-level tracing functions from the DU to the RIC. The interfaces 155 in some embodiments enable the RAN to provide inputs to the functions at the RIC, and, at least in some embodiments, receive outputs that have been computed by these functions at the RIC.
In some embodiments, the MAC control assistor 220 can include various functions such as (1) User Equipment (UE)-specific beamforming weight calculation based on UL SRS channel signal reception, (2) UE Radio Frequency (RF) condition prediction, and (3) Multi-User, Multiple Input, Multiple Output (MU-MIMO) pairing suggestion for the MAC scheduler based on the UE-specific beams. For each of these functions, some embodiments expose a report interface (that provides input data for the function to the RIC from the DU) and a control interface (that provides output data for the function to the DU from the RIC).
The user-level tracer 222, in some embodiments, produces L1/L2/L3 level information related to user configuration and traffic performance. This tracing data can be used as inputs to various control algorithms, including the MAC scheduler, parameter setting, etc. The user-level tracer 222 can include tracing operations that can (i) track user behavior in a cell, (ii) track user RF condition, (iii) track user data traffic performance in different layers (MAC, Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP)), and (iv) track user RF resource consumption.
Each of the components 310-330 of the MAC control assistor 300 includes an uplink and a downlink, as shown. For the UE-specific BWC function, some embodiments expose a report interface for an uplink Sounding Reference Signal (UL SRS) channel response matrix that is an input to the weight calculation function and a control interface for a UE-specific beamforming weight matrix. For the UE RF condition predictor function, some embodiments expose a report interface for a downlink (DL) channel condition report that is an input to the RF condition prediction and a control interface for a predicted DL channel condition (e.g., including DL SINR, PMI, and rank) for the next scheduling window. For the MU-MIMO pairing suggestion function, some embodiments expose a report interface for UE-specific beamforming weight matrix that is an input to the pairing suggestion function and a control interface for UE pairing suggestion and SINR impact assessment.
For these tracing operations, some embodiments expose report interfaces for the DU and/or the CU to provide various metrics to the user level tracing operations. These metrics can include selected RRC messages, MAC/RLC/PDCP traffic volume and performance, RF condition, and RF resource consumption. In some embodiments, messages over these interfaces to the RIC are triggered based on user behavior and/or periodic reporting (e.g., for traffic performance and RF condition/resource consumption).
The tracing operations track the various user data indicated above, and can provide this information either back to the RAN or to other control algorithms (e.g., other algorithms operating at the RIC). For instance, these algorithms might perform analysis on the user data performance from the user level tracing operations, determine that certain performance is inadequate, and modify how the RAN is treating the user traffic. Examples of control algorithms that can benefit from user-level tracing in some embodiments include (1) traffic steering, (2) quality of service (QoS) scheduling optimization, (3) user configuration adjustment, and (4) user behavior anomaly detection.
For all of the operations described in
As shown, the set of services include conflict mitigation services 550, app subscription management services 552, management services 554, and security services 556. Additionally, the set of termination interfaces include O1 termination interface 580 connecting the SMO to the near real-time RIC, A1 termination interface 582 connecting the non-real-time RIC to the near real-time RIC, and E2 termination interface 584 connecting the E2 nodes to the near real-time RIC. Each of the apps, in some embodiments, is representative of the various functions of the RIC that use data sent from the E2 nodes 520. For example, app 530 may correspond to the UE-specific BFWC 310 of the MAC control assistor 300, app 532 may correspond to the UE RF condition predictor 320 of the MAC control assistor 300, etc.
In some embodiments, the objective of the framework 500 is to offload near real-time functions that are computation-intensive, and provide results back to the O-DU (e.g., via the E2 interface with E2 nodes 520). The results, in some embodiments, can be used to assist or enhance the real-time decision in the MAC layer. Three example use-cases for the MAC control assistance framework, each example specific to a different component of the MAC control assistor (e.g., the UE-specific BFWC, the UE RF condition predictor, and the MU-MIMO pairing suggestor), and one use-case example for the user-level tracer, will be described below.
The first example use-case is specific to the UE-specific beamforming weight calculation based on UL SRS signal reception component of the MAC control assistance framework (e.g., component 310 of the MAC control assistor 300). In some embodiments of this use-case, the input metrics can include multiple options based on UL SRS, such as raw SRS received data, and an SRS channel responses matrix from a channel estimate.
The algorithm for producing output metrics, in some embodiments, evaluates the optimal beam-forming weights to reach the user. Some embodiments use traditional signal processing algorithms that are based on channel models. Alternatively, or conjunctively, machine-learning based algorithms that utilize raw data inputs are used, which require feedback from the DU in the E2 nodes 520.
In some embodiments, the output metrics resulting from the algorithm include a beam-form weight (BFW) matrix for the user. In some embodiments, the BFW could also be mapped to a beam index from a pre-designed beam set. The DU in some embodiments uses the matrix to control the MIMO antenna array gain/phasing in the RU (e.g., the O-RU 135 in the architecture 100) for user data transmission and reception.
The second use-case example is specific to the UE RF condition predictor component of the MAC control assistance framework (e.g., component 320 of the MAC control assistor 300). For this second use-case, the input metrics include at least a channel report from the UE, such as Wideband or Subband CQI/PMI/RI for DL, or SRS for UL, according to some embodiments. The input metrics of some embodiments can also opt to include supportive information such as UE distance, UE positioning, etc.
In some embodiments, the app algorithm for this second use-case is meant to predict the UE’s RF condition based on the observation. Some embodiments utilize traditional signal processing algorithms based on channel and mobility models. Alternatively, or conjunctively, some embodiments also use machine learning based algorithms using data inputs and potentially other factors, such as site layout (which requires feedback from the DU).
The output metrics for this use-case, in some embodiments, include the predicted channel condition of the user for the next scheduling window, as well as predicted downlink and uplink SINR, a precoding matrix (e.g., if applicable), and SU-MIMO layers. In some embodiments, these output metrics are used by the DU for the user link adaptation on PDCCH/PDSCH/PUSCH transmissions.
The third use-case example is specific to the MU-MIMO pairing suggestor to MAC scheduler component (e.g., component 330 of the MAC control assistor 300). The input metrics for this example use case, in some embodiments, include at least the UE-specific BFW matrix and the UE RF condition estimate. Some embodiments may also include supportive metrics such as user data demand, etc., as input metrics in addition to the UE-specific BFW matrix and the UE RF condition estimate.
The app algorithm for this use-case, in some embodiments, is meant to identify users that can be paired for MU-MIMO operations. For example, some embodiments of the third use-case use traditional signal processing algorithms based on information theory and cross-channel covariance evaluation. Alternatively, or conjunctively, some embodiments use machine learning based algorithms using the data inputs, which again requires feedback from the DU.
In some embodiments, the output metrics of this third use-case can include UE pairing suggestions and an impact assessment on SINR and SU-MIMO layers. Additionally, the DU in some embodiments uses the output metrics to select users for RF scheduling, and to determine the transmission efficiencies.
An example use-case for the user-level tracer can include QoS scheduling optimization with the goal of adjusting a user’s scheduling priority for an RF resource to optimize the service quality. The input for some embodiments of this use-case can include a service quality target from a user subscription. In some embodiments, the user-level tracing includes (1) tracking the user RF condition, (2) tracking the user data traffic performance in different layers (e.g., MAC/RLC/PDCP), and (3) tracking the user RF resource consumption.
In some embodiments, the app algorithm is based on the QoS target and observed user traffic performance, and can be used to determine that a user’s resource allocation is insufficient. The algorithm format, in some embodiments, can be logic-based or machine learning-based. In some embodiments, the output can include a recommendation issued to the MAC scheduler to adjust the traffic priority or link adaptation in order to improve performance.
On each machine (e.g., each VM or Pod) that executes a control plane application, some embodiments configure a RIC SDK to serve as an interface between the control plane application on the machine and a set of one or more elements of the RAN. In some embodiments, the RIC SDK provides a set of connectivity APIs (e.g., a framework) through which applications can communicate with the distributed near real-time (RT) RIC implemented by two or more near real-time RICs. Examples of such applications include xApps, and other control plane and edge applications in some embodiments. In O-RAN, xApps perform control plane, monitoring and data processing operations. The discussion below regarding
The control plane application on each machine communicates with the set of RAN elements through high-level APIs 620 that the RIC SDK converts into low-level APIs 625. In some embodiments, at least a subset of the low-level API calls 625 are specified by a standard specifying body. Also, in some embodiments, the high-level APIs 620 are made in a high-level programming language (e.g., C++), while the low-level API calls comprise low-level calls that establish and maintain network connections and pass data packets through these connections.
The set of RAN elements that the RIC SDK connects with the control plane application on its machine in some embodiments include RAN elements that are produced and/or developed by different RAN vendors and/or developers. These RAN elements include CUs 630 and DUs 635 of the RAN in some embodiments. Also, this SDK communicates with the CUs and DUs through the low-level, standard-specified E2 interface, while the control plane application on the machine uses high-level API calls to communicate with the CUs and DUs through the RIC SDK. In some embodiments, the high-level API calls specifying E2 interface operations at a high-level application layer that do not include low-level transport or network operations.
Conjunctively, or alternatively, the set of RAN elements that the RIC SDK connects with the control plane application 615 on its machine 610 include network elements of the RIC. Again, these network elements in some embodiments include RAN elements that are produced and/or developed by different RAN vendors and/or developers. These RIC elements in some embodiments include shared data layer (SDL) 560, datapath input/output (I/O) elements, and application and management services 552 and 554 in some embodiments.
Through the distributed near RT RIC, the RIC SDK also connects its control plane application to other control plane applications executing on other machines. In other words, the RIC SDK and the distributed near RT RIC in some embodiments serve as communication interface between the control plane applications. In some embodiments, the different control plane applications are developed by different application developers that use the common set of RIC APIs to communicate with each other through the distributed near RT RIC. In some of these embodiments, the distributed near RT RIC adds one or more parameters to the API calls as it forwards the API calls from one control application to the other control application.
The API calls from second CP application 920 to the first CP application 915 are forwarded through the second RIC SDK 904, the second RIC 901, the first RIC 900, and the first RIC SDK 902, while responses to these API calls from the first CP application 915 to the second CP application 920 are forwarded through the first RIC SDK 902, the first RIC 900, the second RIC 901 and the second RIC SDK 904.
For each of these E2, A1, and O1 APIs, the RIC SDKs 1215 provide high-level counterpart APIs for the control plane applications 1220 that use the RIC SDKs and the distributed near RT RIC platform to communicate with the E2 nodes 1202-1206, the non-real-time RIC platform 1208 and the SMO 1210.
Enablement APIs are the APIs that are used in some embodiments to allow the control plane applications 1220 to communicate with each other. As described above by reference to
The enablement APIs in some embodiments include registration APIs, service discovery APIs as well as inter-app communication APIs. Registration APIs are used by the applications 1220 (e.g., xApps) to introduce themselves to other applications 1220 by providing their network identifiers (e.g., their network address and available L4 ports) and providing their functionality (e.g., performing channel prediction). Service discovery APIs allow control plane applications 1220 (e.g., xApps) to query the service directory (e.g., of the distributed near RT RIC) for other control plane applications (e.g., other xApps) that provide a particular service. The inter-app communication APIs allow the control plane applications to communicate with each other to pass along data and/or request certain operations.
Some embodiments deploy an SDL cache on the same host computer as a control plane application, and use this cache to process at least a subset of the SDL storage access requests of the control plane application. In some embodiments, the control plane application and the SDL cache operate on a machine that executes on the host computer. In other embodiments, the SDL cache operates on the same host computer but outside of the machine on which the control plane application executes. In some of these embodiments, multiple control plane applications executing on the same host computer use a common SDL cache on that host computer.
The SDL cache is part of a RIC that executes on the same host computer as the control plane application in some embodiments. In other embodiments, the SDL cache is part of the RIC SDK that executes on the same machine as the control plane application. In either of these embodiments, a synchronizing process of the RIC or the RIC SDK synchronizes the data stored in the SDL cache with the data stored in the SDL storage.
In some embodiments, the SDL storage operates on a different host computer than the host computer on which the control plane application executes, while in other embodiments at least a portion of the SDL storage operates on the same host computer on which the control plane application executes. Also, in some embodiments, the RIC or the RIC SDK forwards SDL access requests from the control plane application to the SDL storage when the RIC SDK cannot process the SDL access requests through the SDL cache. For instance, the RIC or the RIC SDK cannot process SDL access requests through the SDL cache when the SDL cache does not store data requested by the control plane application.
When the control plane application 1310 uses a high-level API call to read or write data to the SDL storage, the query manager 1325 of the RIC SDK 1300 first determines whether the data record being read or written is stored in the SDL cache 1302. If so, the query manager 1325 reads from or write to this record. When this operation is a write operation, the synchronizing service 1327 writes the new data in real-time or on batch basis to the SDL storage 1350. On the other hand, when query manager 1325 of the RIC SDK 1300 determines that the data record being read or written is not stored in the SDL cache 1302, it passes the API call to the SDL layer of the distributed near RT RIC to perform the requested read or write operation. When passing this API call, the RIC SDK 1300 modifies the format of this call and/or modifies the parameters supplied with this call in some embodiments.
Some embodiments provide various methods for offloading operations in an O-RAN (Open Radio Access Network) onto control plane (CP) or edge applications that execute on host computers with hardware accelerators in software defined datacenters (SDDCs). For instance, at the CP or edge application operating on a machine executing on a host computer with a hardware accelerator, the method of some embodiments receives data, from an O-RAN E2 unit, for which it has to perform an operation. The method uses a driver of the machine to communicate directly with the hardware accelerator to direct the hardware accelerator to perform a set of computations associated with the operation. This driver allows the communication with the hardware accelerator to bypass an intervening set of drivers executing on the host computer between the machine’s driver and the hardware accelerator. Through this driver, the application in some embodiments receives the computation results, which it then provides to one or more O-RAN components (e.g., to the E2 unit that provided the data, to another E2 unit or to another xApp).
In some embodiments, a Pod is a small deployable unit of computing that can be created and managed in Kubernetes. A Pod includes a group of one or more containers with shared storage and network resources, and a specification for how to run the containers. In some embodiments, a Pod’s contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific logical host computer; it contains one or more application containers that are communicate with each other. In some embodiments, the shared context of a Pod is a set of an operating system namespaces (e.g., Linux cgroups). Within a Pod’s context, the individual applications may have further sub-isolations applied.
Each Pod’s accelerator driver 1412 has direct accesses to the hardware accelerator 1450, and this access bypasses the hardware accelerator drivers 1414 and 1416 of the VM 1406 and the hypervisor 1408. In some embodiments, the hypervisor 1408 executes over an operating system (not shown) of the host computer 1410. In these embodiments, the direct access of each Pod’s accelerator driver 1412 to the hardware accelerator 1450 also bypasses the hardware accelerator driver of the operating system.
To communicate with the hardware accelerator, each application 1402 in some embodiments communicates through the RIC SDK 1430 executing on its Pod. For instance, in some embodiments, each application 1402 uses high-level APIs of the RIC SDK 1430 to communicate with the hardware accelerator 1450. The RIC SDK 1430 then converts the high-level APIs to low-level APIs that are needed to communicate with machine’s driver 1412, which, in turn, relays the communication to the hardware accelerator 1450. The low-level APIs are provided by a first company associated with the sale of the hardware accelerator 1450, while the RIC SDK 1430 is provided by a second company associated with the distribution of the RIC SDK 1430. In some embodiments, the low-level APIs used by the RIC SDK 1430 are APIs specified in an API library 1432 associated with the hardware accelerator 1450.
As shown in
The application 1402 receives (at 1505) the data from the E2 unit 1650 through (1) the distributed near RT RIC 1680 formed by near RT RICs 1640 and 1645 executing on host computers 1410 and 1610, and (2) the RIC SDK 1430 executing on its Pod 1404. The application 1402 then uses (at 1510) the hardware accelerator 1450 to perform a set of computations associated with the operation.
To communicate with the hardware accelerator 1450, the application 1402 uses high-level APIs provided by the RIC SDK 1430. The RIC SDK 1430 then converts the high-level APIs to low-level APIs specified in the API library 1432 associated with the hardware accelerator 1450. These low-level APIs are then communicated to the hardware accelerator 1450 by the Pod’s driver 1412 through its direct, passthrough access to the accelerator 1450, which bypasses the drivers 1414 and 1416 of the VM 1406 and hypervisor 1408. Through this driver 1412, the APIs specified in the API library 1432, and the RIC SDK 1430, the application 1402 also receives the results of the operations (e.g., computations) performed by the hardware accelerator 1450.
The application 1402 provides (at 1515) the result of its operation to one or more O-RAN components, such as the E2 unit 1650 that provided the data that started the process 1500 or the SDL storage. This result is provided through the RIC SDK 1430 and the distributed near RT RIC 1680. In other embodiments, the application 1402 (through the RIC SDK 1430) provides the results of its operation to one or more other applications (applications other than the E2 unit that provided the data for which the application performed its operation) operating on another O-RAN E2 unit or machine executing on the same host computer or on another host computer as the application that uses the hardware accelerator 1450 to perform the operation. The process 1500 ends after 1515.
Other embodiments use the passthrough access for the O-RAN control or edge application in other deployment settings. For instance,
To use the hardware accelerator 1750, each application 1702 in some embodiments uses high-level APIs of the RIC SDK 1730 (executing on its Pod 1704) to communicate with the hardware accelerator 1750. The RIC SDK 1730 converts the high-level APIs to low-level APIs that are needed to communicate with VM’s driver 1712, which, in turn, relays the communication to the hardware accelerator 1750. In some embodiments, the low-level APIs used by the RIC SDK 1730 are APIs specified in an API library 1732 associated with the hardware accelerator 1750. This API library 1732 is part of the driver interface of the VM 1706.
The VM’s accelerator driver 1812 bypasses the hardware accelerator drivers 1816 of the hypervisor 1806. In some embodiments, the hypervisor 1806 executes over an operating system (not shown) of the host computer 1810. In these embodiments, the direct access of the VM’s accelerator driver 1812 to the hardware accelerator 1850 bypasses the hardware accelerator driver of the operating system.
To use the hardware accelerator 1850, each application 1802 in some embodiments uses high-level APIs of the RIC SDK 1830 (executing on its Pod 1804) to communicate with the hardware accelerator 1850. The RIC SDK 1830 converts the high-level APIs to low-level APIs that are needed to communicate with the VM’s driver 1812, which, in turn, relays the communication to the hardware accelerator 1850. In some embodiments, the low-level APIs used by the RIC SDK 1830 are APIs specified in an API library 1832 associated with the hardware accelerator 1850. This API library 1832 is part of the driver interface of the VM 1806.
One of ordinary skill will realize that the passthrough access for the O-RAN control or edge application is used in other deployment settings in other embodiments. For instance, instead of operating on Pods, the applications in other embodiments operate on containers. These embodiments then use the hardware accelerator drivers of their Pods or VMs to have passthrough access to the hardware accelerators for the control or edge application. In some of these embodiments, the control or edge application communicates with the hardware accelerator through its associated RIC SDK, and communicates with other O-RAN components (to receive data and to provide results of its processing of the data) through its associated RIC SDK and the distributed near RT RIC connecting the O-RAN components and the application. In some embodiments, the control or edge application in these embodiments performs processes similar to process 1500 of
The above-described direct, passthrough access to hardware accelerators is quite beneficial for O-RANs. The RIC is all about decoupling the intelligence that used to be embedded within the RAN software (CU and DU) and moving it to the cloud. One benefit of this is to use more advanced computing in the cloud for the xApp and edge operations (e.g., for ML, deep learning, reinforcement learning for control algorithms, etc.). A DU close to a cell site typically cannot run advance computations because it would not be economically feasible to put GPUs at each cell site as network cap X will be very high.
By using the hardware accelerator (GPU, FPGAs, eASICs, ASICs) in the SDDC, some embodiments run complex control algorithms in the cloud. Examples of such xApps include Massive MIMO beam forming and Multi-user (MU) MIMO user pairing, which were described above. Generally, any xApp whose computations can benefit from massive parallelization would gain the benefit of GPU or other accelerators. The use of ASICs is beneficial for channel decoding/encoding (turbo encoding, LDPC encoding, etc.). In some embodiments, the RIC is typically on the same worker VM as xApps. However, in other embodiments, the RICs executes on a different host computer so that more xApps that need GPUs and other hardware accelerators can run on the hosts with the GPUs and/or other hardware accelerators.
The process 1900 uses (at 1910) the set of installation files to configure, based on the description relating to the passthrough access, a program executing on the host computer to pass calls from a particular hardware accelerator driver associated with the application to the hardware accelerator without going through an intervening set of one or more drivers for the hardware accelerator that executes on the host computer between the particular hardware accelerator driver and the hardware accelerator. This configuration allows the application to bypass the intervening set of drivers when directing the hardware accelerator to perform operations for the application and to receive the results of the operations from the hardware accelerator.
The program that is configured at 1910 in some embodiments is the host’s operating system, while in other embodiments it is a hypervisor executing on the host computer. In still other embodiments, the program is a virtual machine (VM) and the application operates on a Pod or container that executes on the VM. The process 1900 completes (at 1915) the installation of the application by processing the remaining set of installation files selected at 1905, and then ends. In other embodiments, the process 1900 performs the configuration of the program as its last operation instead of as its first operation at 1910. In still other embodiments, it performs this configuration as one of its intervening installation operations.
Before performing the selection and configuration, the deployment process of some embodiments identifies the host computer from several host computers as the computer on which the application should be installed. The process in some embodiments identifies the host computer by determining that the application requires a hardware accelerator, identifying a set of host computers that each comprise a hardware accelerator, and selecting the host computer from the set of host computers. The process selects the host computer by (1) determining that the application will need to communicate with a set of one or more other applications that execute on the selected host computer, and (2) selecting the host computer as the set of other applications simultaneously executes on the host computer. This installation of the application with the set of other applications on the selected host computer reduces communication delay between the application and the set of other applications.
Some embodiments have the hardware accelerator drivers of the O-RAN control or edge applications communicate with virtualized hardware accelerators that are offered by an intervening virtualization application (e.g., hypervisor) that executes on the same host computer as the application. For instance, the method of some embodiments deploys a virtualization application on a host computer for sharing resources of the host computer among several machines executing on the host computer. This computer has a first set of one or more physical hardware accelerators.
The method deploys several applications on several machines to perform several O-RAN related operations for a set of O-RAN components. Through the virtualization application, the method defines a second set of two or more virtual hardware accelerators that are mapped to the first set of physical hardware accelerators by the virtualization application. The method assigns different virtual hardware accelerators to different applications. The method also configures the applications to use their assigned virtual hardware accelerators to perform their operations.
In some embodiments, the deployed machines are Pods, and the applications are deployed to execute on the Pods. At least two Pods execute on one VM that executes above the virtualization application. This VM includes a hardware accelerator driver that is configured to communicate with two different virtual hardware accelerators for the two applications executing on the two Pods. In other embodiments, multiple Pods execute on one VM that executes above the virtualization application, and each Pod has a hardware accelerator driver that is configured to communicate with a virtual hardware accelerator that is assigned to that driver.
Each Pod’s accelerator driver 2012 has direct access to the virtual accelerator 2052 or 2054, and this access bypasses the accelerator drivers 2014 and 2016 of the VM 2006 and the hypervisor 2008. In some embodiments, the hypervisor 2008 executes over an operating system (not shown) of the host computer 2010. In these embodiments, the direct access of each Pod’s accelerator driver 2012 to the virtual accelerator 2052 or 2054 also bypasses the hardware accelerator driver of the operating system.
As shown, the virtual accelerators 2052 and 2054 communicate to the hardware accelerator 2050 through the accelerator manager 2060 of the hypervisor 2008. The accelerator manager 2060 allows the virtual accelerators 2052 and 2054 (and in turn their associated applications 2002) to share one hardware accelerator 2050, while operating with this accelerator 2050 as if it is dedicated to their respective applications and Pods 2002 and 2004. Examples of such a hardware accelerator 2050 include a graphical processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a structured ASIC.
To communicate with its virtual accelerator 2052 or 2054, each application 2002 in some embodiments communicates through the RIC SDK 2030 executing on its Pod 2004. For instance, in some embodiments, each application 2002 uses high-level APIs of the RIC SDK 2030 to communicate with its virtual accelerator 2052 or 2054. The RIC SDK 2030 then converts the high-level APIs to low-level APIs that are needed to communicate with each machine’s driver 2012, which, in turn, relays the communication to the virtual accelerator 2052 or 2054. The virtual accelerator 2052 or 2054 then relays the communications to the hardware accelerator 2050 through the accelerator manager 2060.
As mentioned above by reference to
To provide a low latency near RT RIC, some embodiments separate the RIC’s functions into several different components that operate on different machines (e.g., execute on VMs or Pods) operating on the same host computer or different host computers. Some embodiments also provide high speed interfaces between these machines. Some or all of these interfaces operate in non-blocking, lockless manner in order to ensure that critical near RT RIC operations (e.g., datapath processes) are not delayed due to multiple requests causing one or more components to stall. In addition, each of these RIC components also has an internal architecture that is designed to operate in a non-blocking manner so that no one process of a component can block the operation of another process of the component. All of these low latency features allow the near RT RIC to serve as a high speed IO between the E2 nodes and the xApps.
Each of the three RIC Pods 2105, 2110, and 2115 communicates with one or more xApp Pods 2120. In some embodiments, each Pod (2105, 2110, 2115 or 2120) is allocated hardware resources (e.g., CPUs, memory, disk storage, network IO, etc.) per the Pod’s unique needs (i.e., per the datapath, service and storage operations performed by each Pod). Also, in some embodiments, each Pod has its own high availability and lifecycle update configuration that matches the unique needs of each Pod.
The service Pod 2110 performs xApp onboarding, registration, FCAPS (fault, configure, accounting, performance, security), and other services in some embodiments. For instance, in some embodiments, the service Pod 2110 provides the management services 554 of the near RT RIC, and performs the O1 termination 580 and the A1 termination 582 to the SMO and its associated non-RT RIC. In some embodiments, each of these components 554, 580 and 582 operate on a separate container in the service Pod 2110, while in other embodiments two or more of these components operate on one container in the service Pod 2110.
As mentioned above, the A1 Interface is between the near-RT RIC and the non-RT RIC in some embodiments. Through this interface, the near RT RIC relays relevant network information as reported by E2 nodes (e.g., CUs and DUs), and the non-RT RIC provides control commands for the E2 nodes (e.g., for control use-cases operation in non-RT granularities). The O1 Interface is between the near-RT RIC and the SMO, and in some embodiments is used for discovery, configuration, resource management and auto-scaling, life-cycle management, and fault tolerance.
The RIC management services 554 in some embodiments include services that the near RT RIC provides to the xApps and to the other RIC components. Examples of the services provided to the xApps include xApp service registry/directory (which the xApps can use to identify other xApps associated with the distributed near RT RIC and the operations performed by these other xApps), and FCAP operations, such as metric collection, policy provisioning and configuration. In some embodiments, the xApps can query the service registry/directory to identify other xApps or other xApps that perform particular services, and can register to receive notifications regarding xApps and their capabilities when the xApps are added to the directory.
Examples of FCAP operations performed by the service Pod 2110 for the xApp include fault operations involving collecting metrics that monitor CPU and memory utilizations to analyze to raise alerts, configuration operations to configure or re-configure the xApps, accounting operations to collect data needed for accounting and performance operations to collect metrics from xApps to analyze to quantify the xApp performance.
For the other RIC components (e.g., the datapath Pod 2105 and the SDL Pod 2115), the service Pod 2110 performs services as well, such as metric collection, policy provisioning and configuration. The service Pod 2110 can be viewed as a local controller that performs operations at the direction of a central controller, which is the SMO. Through the SMO, the service Pod 2110 would receive configuration and policies to distribute to the xApps and the other RIC components. Also, to the SMO, the service Pod 2110 provides metrics, logs and trace data collected from the xApps and/or RIC components (e.g., the datapath Pod and the SDL Pod). In some embodiments, the service Pod can be scaled (e.g., replicated) and backed up independently of the other Pods. In some embodiments, the service Pod has a data cache that is a cache for a time series database of the SMO. In this cache, the service Pod stores stats, logs, trace data and other metrics that it collects from the xApps and one or more RIC components before uploading this data to the SMO database.
The SDL Pod 2115 implements the SDL 560 and its associated database 570. As further described below, the SDL Pod 2115 in some embodiments also executes one or more service containers to execute one or more preprocessing or post-processing services on the data stored in the SDL. Like the service Pod, the SDL Pod in some embodiments can be scaled (e.g., replicated) and backed up independently of the other Pods.
The datapath Pod 2105 includes several important near RT RIC components. These are the E2 termination 584, the conflict mitigation 550, the application subscription management 552, and the RIC SDK interface 2150. As further described below, some or all of these datapath services in some embodiments are embedded in a datapath thread and a control thread of the datapath Pod. In other embodiments, the datapath services are embedded in a data IO thread, multiple data processing threads and a control thread.
A thread is a component of a process that executes on a computer. The process can be an application or part of a larger application. A thread is a sequence of programmed instructions that can be managed independently of other threads of the process. Multiple threads of a given process can execute concurrently (e.g., by using multithreading capabilities of a multi-core processor) while sharing the memory allocated to the process. Multithreading is a programming and execution model that allows multiple threads to exist within the context of one process. These threads share the process’s resources, but are able to execute independently.
The control thread in some embodiments is the interface with the service Pod and SDL Pod for the datapath threads, while in other embodiments it is the interface to just the service Pod for the datapath threads (as the datapath threads can communicate directly with the SDL Pod). The control thread in either of these approaches performs the slower, control related operations of the datapath, while the one or more datapath threads perform the faster IO operations of the datapath. The control thread in some embodiments interfaces with the service Pod to receive configuration data for configuring its own operations as well as the operations of the datapath thread.
The embodiments that separate the datapath thread into a data IO thread and multiple data processing threads further optimize the data IO by pushing the more computationally intensive operations of the datapath thread into multiple datapath processing threads, which then allows the less computationally intensive operations to run in the data IO thread. Both of these optimizations are meant to ensure a fast datapath IO (one that does not experience unwanted latencies) so that the near RT RIC can serve as a high speed interface between E2 nodes 2118 and xApps 2120.
As mentioned above, the Pods 2105, 2110 and 2115 communicate through high speed inter Pod interfaces. In some embodiments, the Pod-to-Pod connections are established through SCTP (streaming control transport protocol) or through even higher speed shared memory (shmem) connections. In some embodiments, the shared memory connections are employed only between a pair of Pods that executes on the same host computer. Examples of such pairs of Pod include (1) a datapath Pod and an SDL Pod, (2) a datapath Pod and a service Pod, (3) a service Pod and an SDL Pod, (4) an xApp Pod and a datapath Pod, (5) an xApp Pod and an SDL Pod, etc. The shared memory is lockless and access to it is non-blocking in some embodiments. Other embodiments use slower interfaces (e.g., gRPC) between the service Pod 2110 and the other Pods 2105, #115, and 2120 as the service Pod is not as critical a Pod as the other Pods.
The different Pods (e.g., 2105, 2110 and 2115) of a near RT RIC in some embodiments can execute on the same host computer or can execute on two or more host computers. In other embodiments, one or more of the Pods (e.g., the service Pod 2110) always operates on a separate host computer than the other Pods (e.g., the datapath Pod 2105 and the SDL Pod 2115). Also, in some embodiments, the Pods 2105, 2110 and 2115 operate on one host computer 2205 along with one or more xApp Pods 2220a, while other xApp Pods 2220b operate on other host computers 2210, as shown in
For instance,
Also, although several near RT RICs are described above and below as being implemented with Pods, the near RT RICs in other embodiments employ VMs to implement the RIC components. Moreover, even in the embodiments that implement the different RIC components with Pods, some or all of the Pods operate on VMs, such as lightweight VM (e.g., Photon VMs provided by VMware, Inc.).
In addition to using fast communication interfaces between the Pods, some or all of the Pods use non-blocking, lockless communication protocols and architectures in some embodiments. For instance, the datapath Pod 2105 uses non-blocking, lockless communication between threads and processes that make up this Pod. Datapath Pod 2105 also uses non-blocking, lockless communication when communicating with the service Pod 2110, the SDL Pod 2115 or the xApp Pods 2120. Non-blocking communication ensures that no first component that sends a request to a second component can stall the operations of the second component when the second component is processing too many requests. In such cases, the second component will direct the first component to resend its request at a later time. The datapath Pod employs lockless communication in that it uses single thread processing that does not employ thread handoffs. Hence, no portion of memory has to be locked to ensure that another process thread does not modify it in an interim time period.
The communication interface between the RIC SDK interface 2150 of the datapath Pod 2105 and the RIC SDK 2112 of an xApp Pod 2120 is also novel in some embodiments. In some embodiments, this interface parses the header of E2AP messages received from E2 nodes, stores some or all of the parsed components in a new encapsulating header that encapsulates the E2SM payload of the E2AP message along with some or all of the original E2AP header. In doing this encapsulation, the SDK interface 2150 in some embodiments performs certain optimizations, such as efficiently performing data packing to reduce message size overhead for communications from one Pod to another (e.g., reduces the size of the E2 Global ID value, etc.). These optimizations improve the efficiency of the near RT RIC datapath Pod and xApp Pod communication.
The near RT RIC in other embodiments has one or more other Pods. For instance,
In some embodiments, the LCM Pod 2405 uses different upgrade methodology to upgrade the different Pods. For instance, the LCM Pod in some embodiments replicates the SDL data store and seamlessly transitions from an active data store to another standby datastore in order to perform a hitless upgrade of the SDL. On the other hand, to upgrade the datapath Pod, the LCM Pod’s procedure is more involved, as it configures the active and standby datapath Pods to be dual-homed connections for each E2 node and each xApp, and configures the active datapath Pod to replicate state with the standby datapath.
The datapath thread 2507 provides a fast datapath IO of the near RT RIC between the E2 nodes 2518 and the xApps 2520. The data plane capabilities of the RIC in some embodiments can be scaled up by implementing the RIC datapath IO with one control thread and multiple datapath threads that share the load for the datapath processing of the datapath Pod. Several such implementations will be further described below. The control thread 2509 performs several control operations associated with the RIC’s datapath. The near RT RIC 2500 separates the control and datapath threads because the data IO operations need to be fast and should not be slowed down by control operations that can operate at a slower rate. In some embodiments, the control and datapath threads are two threads in a single process (i.e., run in the same shared memory address space).
In some embodiments, each of these threads uses non-blocking, lockless interfaces to communicate with other components in this architecture (e.g., with the RIC SDK, service Pod agent, SDL agent, and/or E2 nodes) to the extent that they communicate with these other components. Also, in some embodiments, both threads use minimal OS system calls and run as infinite loops. As further described below, the datapath thread and the control thread exchange data over two circular rings 2522 (called cbuf), with one ring handling messages from the datapath thread to the control thread and the other handling messages from the control thread to the datapath thread.
The control thread 2509 serves as the control interface to the E2 nodes 2518, the SMO 2530 (through the service Pod agent 2524), the xApps (e.g., through SCTP), and the SDL Layer (through the SDL agent 2526). In some embodiments, the control thread is the main thread to communicate with these external entities; however, as further described below, the datapath thread in some embodiments also communicates with the SDL 2515 through the SDL agent 2526.
The control thread 2509 in some embodiments handles all control functions. This thread sends various control parameters to other functions, and in some embodiments enforces admission controls. In other embodiments, the datapath thread 2507 enforces admission controls and the SMO through the service Pod specifies the admission controls. The control thread 2509 in some in some embodiments has control channel communications with the RIC SDK of an xApp Pod through SCTP. In other embodiments, the control thread communicates with the RIC SDK of an xApp Pod through gRPC. Also, in some embodiments, the control thread communicates with the RIC SDK through shared memory (shmem) when the xApp Pods and the datapath Pod execute on the same host computer
The control thread 2509 also provides the transport mechanism to transport the statistics, logs and trace data generated by the datapath thread 2507. In some embodiments, some or all of this data is transported to the SDL Pod 2515 through the SDL agent 2526 and/or to the SMO 2530 through the service agent 2524. The control thread 2509 in some embodiments negotiates security keys with E2 node peers, and passes these keys to the datapath thread, which uses them to perform its encryption/decryption operations.
The datapath thread 2507 provides the high speed IO between E2 nodes and xApps. This thread handles the RIC SDK interface and the E2 termination operations, as well as the conflict mitigation and xApp subscription operations in some embodiments. This thread performs ASN.1 decoding of E2AP messages to extract the message data. In some embodiments, the datapath thread does not decode the E2SM payload of these messages. The datapath thread 2507 validates E2 and xApp messages and sequences. In some embodiments, the message types include E2 node setup and service update, E2 node indication reports, xApp initiated subscriptions for E2 node data and xApp initiated control requests.
The datapath thread 2507 in some embodiments runs E2 state machines in order to create and maintain state on behalf of xApps (e.g., state of E2 nodes, subscriptions to E2 nodes, etc.). Also, in some embodiments, the datapath thread performs table lookups to send messages to xApps that request data. This thread also handles control requests from xApps towards E2 nodes and forwards back responses to these requests from the E2 node to the xApps.
The datapath thread communicates with the xApps through SCTP when the xApps are on another host computer, or through shared memory when the xApps are on the same host computer. In some embodiments, the xApp messages have CRC bits to detect corruption. These messages also carry timestamps and can be compressed in some embodiment. The datapath thread 2507 performs data replication for multiple subscriptions. The datapath thread 2507 also performs datapath security operations, e.g., by signing, encrypting and decrypting data messages.
As mentioned above and further described below, the datapath thread 2507 communicates with the control thread 2509 in some embodiments via a pair of rings 2522. In some embodiments, the frequency of messages between the two threads can be tuned (e.g., can be configured) to be from sub milliseconds to seconds per ring pair. Through the control thread, the datapath thread 2507 receives configuration data updates and state changes. The datapath thread 2507 generates statistics, logs and traces and provides the generated statistics, logs and trace data to the control thread for storage in the SDL and/or to provide to the SMO.
The datapath thread 2507 also performs conflict management operations in case multiple xApps try to set the same parameters to the same E2 node at the same time. For instance, the conflict management operations ensure that two xApps do not try to change a cellular network setting (e.g., a direction of an antenna) differently within a short time period. In some embodiments, the datapath thread’s conflict management employs different methodologies to address different types of conflicts, e.g., (1) for one set of requests, for a duration of time, it rejects a second request to modify a parameter after receiving a conflicting earlier first request, (2) for another set of requests, it rejects a request regarding a parameter from one xApp when another higher priority xApp makes a conflicting request for the same parameter, (3) for another set of requests regarding another set of parameters, it only accepts requests made by xApps that are allowed to make such requests during particular periods of time. The policies for handling these conflicts are provided by the SMO 2530 through the service Pod’s agent 2524.
In
The RIC SDK also uses control communication channel with the service Pod 2510 and the control thread 2509 for xApp control operations such as xApp onboarding, registration, capabilities, subscription, FCAPS, etc. In some embodiments, the control channel communication between the SDK and the control thread 2509 is through shared memory when the xApp Pod (and its SDK) and the datapath Pod operate on the same host computer, and through SCTP when they operate on different computers. Also, in some embodiments, the control channel communication between the xApp Pod (and its SDK) and the service Pod is through shared memory when the SDK and the service Pod operate on the same host computer, and through gRPC when they operate on different computers. Other embodiments use SCTP for communications between the SDK and the service Pod when the xApp Pod (and its SDK) and the service Pod operate on different host computers.
Some of embodiments use proto bufs when the RIC SDK communicates with the service Pod through gRPC. Also, in some embodiments where the RIC SDK’s communication with the datapath Pod is over shared memory, the shared memory communication uses proto bufs. The RIC SDK has APIs for data functions, e.g., E2 messages to and from E2 nodes. These APIs also include control function messaging, such as onboarding xApp (name, version, function), message subscription, keep alive messaging, and A1 and O1 interface communications with the SMO through the service Pod (e.g., communications to store stats, logs, and trace data in a time series database on the SMO or service Pod, such as Prometheus and ELK).
Some embodiments assign the datapath thread and control thread to one processor core, assign the SDL to another processor core (in order to isolate it from data and control threads), and assign the service Pod to yet another processor core. When one or more xApps execute on the same host computer as the RIC, the xApps are assigned to different cores than the RIC Pods, where multiple xApps can be assigned to the same core, or individual cores are assigned to individual xApps, as needed.
To improve the performance of the RIC and the xApps further, other embodiments perform other hardware assignment optimizations, such as particular memory allocations (e.g., larger RAM allocations) and particular IO allocations. Examples of special IO allocations for some of the Pods include (1) SRIOV allocations for an xApp Pod on one host computer to communicate with a datapath Pod on another host computer, (2) SRIOV allocations for a datapath Pod to communicate with E2 nodes, (3) SRIOV allocations for an xApp Pod on one host computer to communicate with a service Pod on another host computer, and (4) gRPC or SCTP communication over the SRIOV allocations, with the gRPC communications having lower bandwidth allocations and being lower priority than the SCTP communications.
In some embodiments, one RIC and several xApps are bundled together to operate on different Pods that operate on one VM. Multiple instances of the RIC can also be deployed in some embodiments with different sets of xApps. Also, in some embodiments, xApps that need to talk to each other are bundled on the same VM.
As mentioned above, some embodiments implement the RIC datapath not as one datapath thread but as one data IO thread along with multiple datapath processing threads. In some embodiments, each datapath processing thread (DPT) is responsible for performing the datapath processing for a different set of E2 nodes, with each E2 node assigned to just one datapath processing thread. In some embodiments, the data IO thread identifies the DPT associated with an E2 message or an xApp message by hashing the E2 node identifier contained in the message and using the hashed value (obtained through the hashing) as an index into a lookup table that provides the DPT identifier of the DPT that needs to process the data message.
Separating the data IO thread 2605 from multiple DPTs 2615 optimizes the data IO of the datapath Pod 2600 by pushing the more computationally intensive operations into the DPTs, which then allows the less computationally intensive IO operations to run in the data IO thread 2605. This optimization ensures a fast datapath IO (one that does not experience unwanted latencies) so that the RIC can serve as a high speed interface between the E2 nodes and the xApps. Also, each E2 node is the responsibility of just one DPT thread 2615, which typically is responsible for several E2 nodes. Because each E2 node is handled by one particular DPT, no two DPTs will try to change one or more records associated with one E2 node. Hence, the datapath Pod 2600 does not need to lock any E2 node’s records as there is clear demarcation of responsibilities vis-a-vis the communications with the E2 nodes.
The data IO thread 2605 performs the following operations (1) managing connections to the E2 nodes and the xApp Pods, (2) transmitting data messages through these connections to and from the E2 nodes and the xApp Pods, (3) performing security operations, (4) control ring communication with control thread 2610 and DPTs 2615, and (5) generating statistics, logs and trace data regarding messages that it processes.
Each DPT thread 2615 performs the following operations (1) message decode and encode operations (e.g., message encrypt and decrypt operations), (2) message validate operations, (3) sequence validate operations, (4) maintain state machine to keep track of state of the E2 node and the xApp requests and subscriptions, (5) perform conflict management, (6) control ring communication with control thread 2610 and DPTs 2615, and (7) generate statistics, logs and trace data regarding messages that it processes.
At 2710, the process 2700 determines whether it has already subscribed to the particular E2 node to receive the particular set of data tuples. This would be the case if the DPT previously sent the particular E2 node one or more subscription requests that individually or collectively requested the particular set of data tuples or a larger set of data tuples that includes the particular set of data tuples.
When the process 27100 determines (at 2710) that it has already subscribed to the particular E2 node to receive the particular set of data tuples, it (at 2715) adds a new record, or updates a record previously specified, for the xApp in this E2 node’s subscription list and specifies in this record the particular set of data tuples that the xApp should receive. After 2715, the process ends.
On the other hand, when the process 27100 determines (at 2710) that it has not already subscribed to the particular E2 node to receive the particular set of data tuples, it has to either send a first subscription to the particular E2 node if it does not have an active subscription with this node, or has to send an updated subscription to the node if it has an active subscription but not one that includes all of the data tuples in the particular set of data tuples specified in the request received at 2705.
Hence, in such a case, the process 2700 (at 2720) adds a new record, or updates a record previously specified, for the xApp in this E2 node’s subscription list and specifies in this record the particular set of data tuples that the xApp should receive. Next, it sends an updated subscription request to the particular E2 node using a previously allocated RIC Request ID. This updated subscription specifies all of the data tuples in the requested particular set of data tuples when none of these tuples were previously requested by an earlier subscription to the particular E2 node, or specifies some of these data tuples when other data tuples in the particular set were previously requested by one or more earlier subscriptions to the particular E2 node. After 2725, the process 2700 ends.
At 2820, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2815) along the cbuf ring 2620 that is for passing messages from the data IO thread to the identified DPT. Next, at 2825, the DPT uses its data structure records (e.g., the records maintained by its state machine) to identify the set of one or more xApps that should get the E2 message. In some embodiments, the identified set of xApps are the xApps that have subscribed to receive data (e.g., all the data or a subset of the data) from the E2 node.
At 2830, the DPT specifies a data message for the data IO thread 2605 to send to the identified set of xApps. This data message is in the encapsulated format described below by reference to Table 1. The DPT then passes (at 2835) the data message to the data IO thread 2605 along the cbuf ring 2620 that is for passing messages from the DPT 2615 to the data IO thread 2605. Next, at 2840, the data IO thread 2605 retrieves the data message from the cbuf ring 2620, identifies the xApps that need to receive the data message, and then sends each identified xApp the data message. After 2840, the process ends.
At 2910, the data IO thread 2605 generates a hash value from the E2 node’s ID. It then uses (at 2915) the hash value as an index into a lookup table to identify the DPT that is assigned to processing messages associated with the E2 node. At 2920, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2915) along the cbuf ring 2620 that is for passing messages from the data IO thread to the identified DPT.
Next, at 2925, the DPT uses its data structure records (e.g., the records maintained by its state machine) to identify the E2 node that should receive the message. In some embodiments, the data message is a subscription request and the identified E2 node is an E2 node to which an xApp wants to subscribe. At 2930, the DPT specifies a data message for the data IO thread 2605 to send to the identified E2 node. This data message is in the E2AP message format required by a standard. The DPT then passes (at 2935) the data message to the data IO thread 2605 along the cbuf ring 2620 that is for passing messages from the DPT 2615 to the data IO thread 2605. Next, at 2940, the data IO thread 2605 retrieves the data message from the cbuf ring 2620, identifies the E2 node that needs to receive the data message, and then sends each identified E2 node the data message. After 2940, the process ends.
In some embodiments, the DPT 2615 might determine that no new subscription message needs to be sent to the E2 node that it identifies at 2925. For instance, before receiving (at 2905) from a first xApp the subscription request for a set of data tuples from an E2 node, the datapath Pod previously sent for a second xApp a subscription request to the same E2 node for the same set of data tuples or for a larger set of data tuples that includes the data tuples requested by the first xApp. In such a case, the DPT 2615 simply adds the first xApp to the subscription list of the E2 node, so that it can provide subsequently received values from the E2 node to the first xApp. In some embodiments, the DPT 2615 also supplies previously received values from the E2 node that are stored in the SDL to the first xApp or directs the xApp to obtain these values from the SDL.
In some cases, the first xApp asks for additional data tuples from the E2 node that the second xApps did not request previously. In such cases, the DPT 2615 would prepare an updated subscription message for the data IO thread to send to the E2 node to request the data tuples that are newly requested by the first xApp. The DPT would also prepare such a message when the second xApp requested additional data tuples from the E2 node after its initial subscription.
In some embodiments, a service Pod 2510 configures the datapath Pod 2600 to instantiate N DPTs when it starts up with N being an integer greater than one. For the datapath Pod 2600 of a near RT RIC, the number N is computed in some embodiments based on the expected number of E2 nodes and xApps that communicate with the E2 nodes through a near RT RIC. The data IO thread 2605 of the datapath Pod 2600 in some embodiments then assigns the E2 nodes to the DPTs based on the order of subscription requests that it receives and the load on the DPTs at the time of these requests.
Next, at 3010, the data IO thread 2605 generates an N-bit hash value from the Global E2 node ID of the particular E2 node, where N is an integer (e.g., is 6 or 8). This N-bit value is used to identify the particular E2 node in a hash LUT (lookup table) as further described below. At 3015, the process 3000 selects a particular DPT for the particular E2 node based on the current load on each of the DPTs of the datapath Pod 2600 (e.g., by selecting the DPT with the least amount of load). In some embodiments, the current load is just based on the number of E2 nodes assigned to each DPT, while in other embodiments the current load is based on the number of E2 nodes and the number of xApp subscriptions to these nodes. In still other embodiments, the current load is computed in other ways.
At 3020, the process 3000 then creates a record in a LUT and in this record associates the N-bit hash value with the identifier of the particular DPT selected at 3015 for the particular E2 node. In some embodiments, the N-bit hash value is an index into the LUT that identifies the record that specifies the particular E2 node’s ID. At 3020, the process 3000 also specifies the state of this record as Active.
At a subsequent time, if the data IO thread encounters a situation where all xApps have canceled their subscriptions to the particular E2 node, the process 3000 maintains the LUT record created at 3020 but changes the status of this record to Inactive. The data IO thread maintains this Inactive status until the next time that an xApp submits a subscription request for the particular E2 node, at which time the status of this record is changed to Active again. This status value is used as a mechanism to ensure that the data IO thread does not have to continuously revisit the E2 node assignments to the DPTs.
Both of these RICs have the same components, which are a datapath Pod 3105, a service Pod 3110, and an SDL Pod 3115. The datapath Pod is shown to include a control thread 3109 and a datapath thread 3107. Instead of one datapath thread 3107, some embodiments employ one data IO thread and multiple DPTs as mentioned above. In some embodiments, the active RIC 3102 is implemented by a first set of one or more computers, while the standby RIC 3104 is implemented by a different second set of one or more computers.
As shown, each E2 node 3118 has a dual-homed SCTP connection with the datapath threads 3107 of the active and standby RICs 3102 and 3104. Similarly, each xApp Pod 3120 has a dual-homed SCTP connection with the datapath threads 3107 of the active and standby RICs 3102 and 3104. Dual-homing connections is a feature provided by SCTP. When a first component connects to an active/standby pair of components through a dual-home connection, the first component can automatically switch to using the standby component when the active component fails. Thus, using the dual-homed SCTP connections, each E2 node or xApp Pod can switch to the datapath thread 3107 of the standby RIC 3104 when the active RIC 3102 or its datapath Pod fails.
As shown, the RIC SDK interface 3122 of the datapath thread 3107 of the active RIC 3102 forwards messages that it receives from the xApp RIC SDKs, and messages that it sends to the xApp RIC SDKs, to the RIC SDK interface 3122 of the datapath 3107 of the standby RIC 3104. This is done in some embodiments so that the standby RIC’s datapath thread 3107 can update its state machine to match the state of the active RIC’s datapath thread 3107. Also, as shown, synchronizing agents 3127 of the active and standby RICs 3102 and 3104 synchronize the SDL storage 3126 of the standby RIC 3104 with the SDL storage 3126 of the active RIC 3102. All components of the active and standby RICs 3102 and 3104 are consistently managed by the SMO 3130.
Also, in some embodiments, the interface 3222 between the near RT RIC and an xApp Pod uses a novel encapsulating header that will be described below by reference to Table 1. The interface 3222 handles a mix of different types of messages. Examples of such messages in some embodiments include (1) the entire E2AP messages (e.g., E2 Setup Request) from an E2 node, (2) some fields of the E2AP header along with the entire E2SM content (i.e., the entire E2AP message payload), (3) internal messages between the near RT RIC and xApp Pod (e.g., a message from the near RT RIC that an earlier message of an xApp caused an error), and (4) messages from xApp to near RT RIC or E2 Node. In some embodiments, the E2 content might not be ASN1 encoded (e.g., portion of a subscription request might not be encoded).
In some embodiments, the near RT RIC 3200 can be configured on a case by case basis to decode just the E2AP messages before it sends the message to an xApp, or to decode the entire E2AP header along with its E2SM payload. In some cases, the near RT RIC sends the entire E2AP header while in other cases it only sends a part of this header. In the RIC E2AP message handling of some embodiments, all fields are in network byte order, and the near RT RIC 3200 will work with that order as much as possible. For displaying fields, some embodiments can convert the data to host order. In some embodiments, the near RT RIC 3200 will not look into E2SM payload, while in other embodiments it will (e.g., in order to avoid duplicate subscription errors).
In some embodiments, the RAN function ID is E2 node specific. The xApps will not subscribe to RAN functions across E2 nodes, as every subscription will be to an individual E2 node. Also, in some embodiments, the RIC Request ID space is local to an E2 node. In some embodiments, the RIC Request ID number space is ephemeral component as well as a persistent component. For example, the RIC request IDs used for indication reports will persist while RIC request IDs used from subscription may be reused.
Table 1 below displays an exemplary message format used in some embodiments for the communication between the RIC and RIC SDK. This is the format of an encapsulating header that is used to encapsulate all messages from and to the RIC to and from the RIC SDK. In some embodiments, the encapsulating header stores data needed by the RIC SDK and the RIC for efficient processing of the data message. In the example illustrated in Table 1, the first sixteen bytes associated with the msg_type, msg_serial_num, msg_len, msg_flags, and ctrl_len are part of the encapsulating header along with the ctrl_info field. The payload of the encapsulated packet can include any data. In the example shown in Table 1, the payload includes the original E2AP packet along with its E2SM payload.
All messages between RIC and RIC SDK are encapsulated with the header shown in Table 1. Control information and payload are optional. Some messages may have control information but no payload field, others might have payload without control information and some may have both control and payload fields. In some embodiments, the RIC SDK can be configured to trap these messages and reformat them for presentation to xApps. The format of the message is a raw byte stream. In some embodiments, a message CRC field is not used, while it is used in other embodiments.
The near RT RIC 3200 handles E2 node and xApp connect, disconnect, reset, and crashes as follows. For E2 nodes, the RIC in some embodiments handles a connect, disconnect, and crash similarly. Specifically, when the connection to the E2 node drops and comes back for any of these reasons, the E2 node will send a connection setup all over again as though it started for the first time, and the near RT RIC will clean all its state related to the E2 node and start over. In some embodiments, the near RT RIC informs all xApps when an E2 node connection drops and comes back up whether they had previously subscribed to the particular E2 node or not, as the E2 node might advertise new functionality in which a previously unsubscribed xApp may be interested. When an xApp connects, disconnects, or crashes, the near RT RIC again performs the same operations, in that it resets all the state of the xApp in the near RT RIC and deletes its subscriptions from all E2 nodes.
After decoding the E2AP message, the near RT RIC creates or updates its internal data structures to account for the received data message, and then creates a flat encapsulated message to the xApp in the format described above by reference to Table 1. As the near RT RIC and RIC SDK operate on different containers and reside on different Pods in some embodiments, they do not pass arbitrary data structures to each other but format their data exchange into an encapsulated message with a specific sequence of bytes in some embodiments. After encapsulating the data message, the near RT RIC forwards the data message to the xApp Pod for the RIC SDK on this Pod to forward to the appropriate xApp.
The internal data structure that the near RT RIC creates or updates while processing the E2AP message is used for processing of responsive messages from the xApp to the E2AP message and for processing of subsequent E2AP messages. Examples of data stored in the near RT RIC’s internal data structure in some embodiments include (1) a subscription list of xApps that are interested in data from a particular E2 node, (2) particular data tuples that each xApp is interested from each E2 node, (3) records identifying network addresses and other location data relating to E2 nodes and xApps, (4) identifiers that are allocated and assigned (e.g., RIC Request IDs).
When the xApp sends a message, its RIC SDK processes the message and forwards it to the RIC along a shared memory or SCTP interface as described above. The near RT RIC then parses the message and stores the parsed components. Based on these components, and on one or more data tuples that it stored in its internal data structure for the associated E2 node message(s), the RIC creates an E2AP response, and then encodes and forwards this response to the E2 node to which it is directed.
For instance, after a first xApp sends a subscription request to receive M data tuples from an E2 node, the near RT RIC’s datapath Pod creates a state to record the first xApp’s desired subscription, requests a subscription with the E2 node for the M data tuples, and forwards these M data tuples to the xApp when it initially receives them and each subsequent time that it receives them. In some embodiments, the near RT RIC’s datapath Pod can be configured to forward the M data tuples to its associated SDL each time that it receives them from the E2 node.
After the first xApp subscribes to receive the M data tuples from the E2 node, a second xApp can subscribe to receive N different data tuples from the E2 node, where N is larger than M. The near RT RIC then sends an updated subscription request to the E2 node. This update now requests the N data tuples. Each time that the near RT RIC receives the N data tuples, it sends M data tuples to the first xApp and all N data tuples to the second xApp.
Another example involves the near RT RIC removing and caching an RIC request ID from an E2AP message from an E2 node in response to a subscription request. After this ID is removed, the RIC provides a portion of the E2AP message and its E2SM payload (if applicable) to the xApp. Subsequently, when the xApp wants to delete the subscription, the RIC retrieves RIC request ID from its state, and inserts it into its E2AP message to the E2 node to request the deletion of the subscription.
In some embodiments, the near RT RIC’s E2 Setup, Response Message, and Failure Message handling is as follows. The near RT RIC initially receives the setup request message from the E2 node. In response, the near RT RIC will decode the message and build internal data structures. The RIC will also cache the raw ASN1 payload. In some embodiments, the near RT RIC accepts all added RAN function identifiers. In some embodiments, the near RT RIC sends the setup message to xApps after decoding the E2AP header but nothing else (i.e., as a message with an ASN1 encoded E2SM payload). In some embodiments, a setup message that the near RT RIC sends to an xApp has a control length (ctrl_len) of 0 with its ASN1 encoded payload.
When an xApp connects later, the near RT RIC will send all setup requests from E2 nodes to the xApp so it has an inventory of connected E2 nodes. In some embodiments, the near RT RIC sends these messages one at a time. Also, as mentioned above, the near RT RIC in some embodiments constructs E2Setup response and send it to the E2 node. In some embodiments, the near RT RIC sends a failure message when a setup request is malformed (e.g., it is a duplicate of the RAN function list, or removes a record not added to a list).
After receiving a reset from E2 node, the near RT RIC performs the following actions after decoding the message. It sends a message regarding this reset to all xApps that have a subscription to this E2 node. In some embodiments, this is an internal message without any ASN1 content. The near RT RIC ends subscription deletion messages to the E2 node for all previous subscriptions that it sent to it. It also sends control, insert and policy deletions to this E2 node. It cleans up any outstanding requests, and sends reset response to the E2 node.
The near RT RIC also employs a Service Update, Acknowledge, and Failure message in some embodiments. This message updates the supported RAN function list, with additions, modifications and deletions. The near RT RIC inform all xApps about the new service configuration of an E2 node. In some embodiments, the near RT RIC sends the message to xApps after application of the configuration so it will reflect the final state of the configuration. In other embodiments, the near RT RIC sends the message as is for xApps to compute the delta between the previous and new state of supported RAN functions. In this latter approach, the near RT RIC does not need to ASN1 encode the resulting delta.
The handling of the E2AP subscription is as follows in some embodiments. An xApp formats the E2SM portion of the subscription and ASN1 encode it. Table 2 below details the control portion of the subscription message (i.e., the portion that is stored in the control field of a message from the xApp to the near RT RIC). The payload will be the ASN1 encoded E2SM content. Multiple subscription message types are defined in some embodiments to disambiguate optional information. Also, in some embodiments, message flags are used to specify the exact format. In some embodiments, each subscription message specifies one E2 node Global ID and one RAN Function ID.
In some embodiments, the E2 node sends an identifier that is 113 bytes and the RIC compresses that to a 40 byte ID. When sending the subscription message to the E2 node, the RIC converts the 40 byte to 113 byte ID. The subscription message control fields will be of fixed formats as far as possible. In some embodiments, the RIC caches all subscription requests and compares requests from multiple xApps in order to avoid sending out duplicate subscription messages. However, when a second subsequent xApp requests additional information from the same E2 node after a first initial xApp requests some information from the E2 node, the RIC resends the subscription (with the same RIC Request ID in some embodiments) but this time asks for the additional information. When sending out the subscription request to the E2 node, the RIC sends out as the E2AP message payload the entire payload received from xApp in some embodiments.
The near RT RIC handles an E2AP RIC subscription response by storing the E2 node Global ID and RIC Request ID (generated by RIC) as control information and sending the exact ASN1 encoded message from E2 node back to xApps.
The E2AP RIC subscription delete request, response or failure message are sent from an xApp to the near RT RIC with message fields sent as control information (i.e., as part of the ctrl_info). The near RT RIC creates the encoded ASN1 message and sends it to the E2 node. The deletion request does not specify the E2 Node Global ID. Hence, this information is provided by the RIC for the xApp. The response message is sent as packed bytes (not ASN1 encoded) from near RT RIC to xApp in some embodiments.
An E2 node’s E2AP indication report is handled as follows. The message is decoded by near RT RIC to determine the RIC Request ID field. This helps determine which xApp(s) subscribed to the indication. The near RT RIC in some embodiments sends the message as an ASN1 encoded to xApp(s). In some embodiments, the near RT RIC also sends the reduced E2 Global ID as control information along with the message.
The near RT RIC’s processing of the E2AP control request is as follows in some embodiments. The xApp sends this request as packed byte information. The near RT RIC does not specify the E2 Global ID in the message, as this information is specified by the xApp. The near RT RIC’s formatting of this message is illustrated in Table 3.
The near RT RIC handles E2AP control response or failure message is as follows. The near RT RIC decodes the message to obtain the RIC Request ID. It then sends the ASN1 encoded message to xApp prepended with the Global E2 Node ID as control information
In some embodiments, the SDL data store is an in memory database that runs in its own set of one or more Pods. It has its own compute and memory resources assigned. As mentioned above, multiple near RT RIC instances define a distributed near RT RIC. In some embodiments, each near RT RIC instance has its own instance of an SDL, which stores system wide information for the RIC instance. Examples of such information include a list of connected E2 nodes (i.e., base station nodes), xApps, subscriptions from each xApp and critical cell data returned by E2 nodes. Further, each SDL instance in some embodiments provides services to preprocess incoming data by running custom algorithms internally as the data arrives and by interfacing to hardware accelerators (e.g., GPUs), or post-process data retrieved from its storage.
The data IO Pod and xApp Pods of the RIC instance are connected to the SDL Pod of the RIC instance. In some embodiments, each SDL instance just operates with the data IO Pod and service Pod of its own RIC instance. Also, in some embodiments, the SDL Pod is managed by the SMO and configured via the service Pod. The dataflows to and from the SDL include (1) data IO to SDL data store, (2) xApps from SDL data store, (3) xApps to SDL data store, (4) data IO from SDL data access (e.g., retrieval of E2 node info, subscription info, etc.), and (5) service Pod to and from SDL communication to provide and retrieve configuration information.
The SDL datapath agent 3414 is the datapath interface that the SDL Pod exposes to the control thread and datapath thread of the datapath Pod 3450. The SDL datapath agent 3414 handles communication from these entities for the SDL, and performs reads and writes to the SDL data store 3420 for these datapath entities. In some embodiments, the SDL datapath agent can be configured to use either SCTP or shared memory libraries to communicate with the datapath Pod 3450.
In some embodiments, the RIC SDK agent 3417 is the agent that the SDL Pod exposes to the RIC SDK of the xApp Pods. The RIC SDK agent 3417 handles communication from the RIC SDKs to the SDL, and performs reads and writes to the SDL data store 3420 for the RIC SDKs. In some embodiments, the RIC SDK agent 3417 can be configured to use either SCTP or shared memory libraries to communicate with the RIC SDKs. This agent 3417 also performs the cache synchronization operation to synchronize the SDL cache of the RIC SDKs with the SDL data store(s) 3420. Also, in cases where it will need to scale to tens of connections from xApps, the connection manager in some embodiments leverages epoll connection handling of the RIC instance (e.g., the epoll connection handling used by the data IO Pod in some embodiments).
The SDL agents 3414 and 3417 handle event subscription and notifications from the RIC SDKs and the datapath Pod. This is separate from E2AP subscription management, but conceptually it is similar. For instance, through this subscription service of the SDL, an xApp specifies its interest in some data via a key and/or frequency of reports. The RIC SDK agent 3417 then provides periodic updates to this xApp based on its subscription. It also provides security services in some embodiments by encrypting and decrypting data.
The data preprocessor and post processors 3416 and 3418 are part of the SDL 3410 in some embodiments in order to flexibly run certain value added algorithms on data. In some embodiments, both of these processors 3416 and 3418 operate in one container, while in other embodiments each of them operates on a separate container in the SDL Pod 3410. Each of these processors also interfaces with external accelerators (e.g., GPUs 3480) to perform their operations in some embodiments. The data preprocessor 3416 runs inline as data is stored in SDL data store 3420.
In some embodiments, the data post processor 3418 runs inline as data is read from the SDL data store 3420. Alternatively, or conjunctively, the post processor 3418 in some embodiments can be configured to run in the background on data stored in the SDL data store 3420 (e.g., to retrieve data from this data store, perform some operations on this data, and store back the results in the data store). The data processors in some embodiments encodes and/or decodes the E2SM payloads of the E2AP messages. This is advantageous as it allows datapath Pod to pass the ASN1 string to SDL to decode and store. As mentioned above, the RIC SDK in some embodiments can also be configured to provide the E2SM encode/decode services.
Another example of a post processor operation that the data processor 3418 performs in some embodiments is a machine trained operation. In some embodiments, the post processor 3418 collects various data tuples stored by various xApps and/or various Pods (e.g., datapath Pod), and passes these data tuples through a machine-trained network (e.g., a neural network trained through machine learning). In some embodiments, to execute its machine-trained network, the post processor 3418 uses one or more hardware accelerators (e.g., one or more GPUs) of the SDL’s host computer to perform its operations. The post processor 3418 access a hardware accelerator through the bypass approach described above by reference to
The post processor 3418 passes the results of its operations to one or more xApps, or it can store the results in the SDL data store 3420 for one or more xApps to retrieve. An example of a result obtained by post-processing SDL data with machine-trained networks includes anomaly detection (e.g., identifying E2 nodes that are behaving anomalously, e.g., cell sites that receive too many connections suddenly).
It is advantageous to use a machine-trained network in the SDL Pod 3410 to process different xApp outputs that are stored in the SDL data store 3420 because this data store 3420 stores the outputs of several xApps as well as numerous data tuples that the E2 nodes provide based on the xApp subscriptions. Often, individual xApps have insight only to the data tuples to which they subscribe and to the results of their own computations and the outputs of a few other xApps. The SDL Pod 3410, on the other hand, has access to a much larger set of E2 node input data and xApp output data. Instead of using machine-trained networks to perform such post-processing, the post processor 3418 uses algorithms (e.g., constrained optimization solvers) to post-process the data stored in the SDL data stores 3420. In other words, the post processor 3418 in some embodiments does not use machine-trained networks but still uses its host computer’s hardware accelerator(s) (e.g., through a bypass path) to perform its operations.
Some embodiments also use post processor 3418 to provide the current state of an E2 node when a first xApp starts to subscribe to the E2 node’s state. After a second xApp subscribed earlier to receive the E2 node’s state, the near RT RIC stores multiple data tuples relating to this node’s state over a duration of time. When the first xApp subsequently subscribes to the E2 node’s state, and either this xApp or the datapath Pod tries to access this state for the first xApp, the post processor 3418 retrieves all the data tuples previously stored for this E2 node in the SDL storage, and uses these data tuples to compute the current state of the E2 node, which it then provides to the first xApp directly or through the datapath Pod.
The SDL data store 3420 is an in-memory database. In some embodiments, an in-memory database is a database that loads into and runs out of the computer system memory (e.g., the host computer volatile memory, including its RAM). One example of such a database is Redis. The data store’s size is selected in some embodiments to minimize search and store latency. Once the existing data store reaches its maximum desirable size, some embodiments create additional instances of this data store in the same or different instances of the SDL.
Also, as shown in
As mentioned above by reference to
In some embodiments, the sources of data to the SDL 3410 include (1) the control thread and datapath thread of the datapath Pod 3450, (2) the xApps through the RIC SDKs, (3) ML model and policy storage and server of the non-RT RIC accessed through the A1 interface, (4) xApp Pod configuration storage, and (5) configuration server of the SMO. In some embodiments, examples of system data (e.g., written by the control thread) in the SDL include (1) E2 node information, (2) cell information, (3) UE information, (4) E2 node reports, (5) KPM (key performance metric) reports, and (6) topology information (from EMS adapters).
Examples of SDL transaction in some embodiments include (1) data IO Pod (control or data IO thread) writing data to SDL, which is then read by an xApp, (2) an xApp reading data from another xApp that is written to the SDL, or the xApp writing data to the SDL for another xApp, (3) xApp writing to SDL so that a service container (e.g., a post processor) operating in the SDL Pod performs an operation (e.g., by using the GPU services or just using the general CPU) on the written data before the same xApp or another xApp retrieves the result of this operation from the SDL, (4) the non-RT RIC reading data from and writing data to the SDL as part of an A1 subscription, (5) SMO storing O1 configuration data in the SDL, and (6) non-RT RIC storing ML data in the SDL.
Some embodiments use CRDs (custom resource definitions) to define resources in an O-RAN system. These resources include applications (xApps, rApps, etc.), components of these applications, deployment constructs, cloud constructs, AI/ML models and RTE constructs. Using these CRD descriptors, app developers can develop and distribute their applications in a platform independent manner. Even though CRDs are routinely used in K8, currently telco companies and providers do not plan on using CRDs to define resources in an O-RAN environment.
The CRDs of some embodiments are used to on board applications into an O-RAN system. In some embodiments, the application on-boarding is secure, portable, developer friendly, follow more standard approaches (e.g., uses an open source approach) as opposed to custom solutions, allow RIC specific customization to the pods, and have an easy to interface with other tools.
In order to run a secure and hardened applications (e.g., xApps or rApps) on dRIC platform, some embodiments enforce the following policies on the applications: (1) network/security policies, (2) POD security policies, (3) limiting resource usages on a cluster, (4) controlling privileges to run, (5) K8s API Authorization (e.g., what resources can be created/deleted/updated), (6) ensuring proper versioning of images being used, and (7) network endpoint configuration (connection to dRIC and connection to other applications).
When CRDs are used to define applications and application components, the SMO in some embodiments gains control over the configuration of the pods deployed on the Kubernetes cluster. All the required controls and policies are enforced in some embodiments using a single layer of automation. This also offers an API endpoint out of the box to perform CRUD (create, read, update, and delete) operations for the applications.
The use of CRDs has several advantages. They allow standardization with new schema that helps to inter-operate SMO and RIC applications. The CRDs provide control over the resource being created, network access etc. Lifecycle management can be owned by SMO. They also allow for policy enforcement at onboarding time. CRDs also enable faster application validation and certification processes. They make application more portable because the same CRD can be implemented in different O-RAN systems and by O-RAN components of different vendors. CRDs also hide implementation of the advanced app configuration (example: SRIOV, shared memory etc.).
To use CRDs, some embodiments use a new controller to process each CRD in the O-RAN system. In some embodiments, the SMO has a new service for processing CRDs. In some of these embodiments, the new service has one controller to process each new CRD. Several examples of new CRDs will now be explained.
Some embodiments include the following new CRDs (1) RANApplicationPackage Spec, (2) RANDeploymentConfiguration Spec, (3) RANDeploymentType Spec, and (4) RANDeployableComponent Spec. In some embodiments, the RANApplicationPackage Spec CRD is as follows:
Any resource type in K8s is mentioned using the “kind” keyword in the CRD and corresponding custom resource. To define a new CRD, a new Custom Resource is expressed as “Kind.” For RAN Application package, the kind is set to RANApplicationPackage. The “apiVersion” field is required and it indicates the stability level of the CRD. It can be updated based on stability or changes to the schema. In the RANApplicationPackage CRD above, the version setting is set to “oran/v1alpha1.”
Every K8s object has the following metadata in a nested object field called “metadata”: (1) name, which is a string that uniquely identifies this object within a namespace and is used in the path when retrieving an individual object, (2) a description, which describes the custom resource, and (3) a UID, which is unique identifier that is unique in time and space value and is used to distinguish between objects with the same name that have been deleted and recreated. In some embodiments, a UID is automatically assigned if not specified. In the RANApplicationPackage CRD example above, the metadata attribute identifies the CRD name as ran-app-pkg-demo, and provides “application package” as its description.
The spec attribute of a CRD indicates the complete description of the CRD’s desired state, which could include the user specified values, system default values or values updated by other components in Kubernetes. There is also system generated field called “Status,” which indicates the current state. In the example above, the specification attribute specifies a configuration field that includes “name,” which is a list of deployment configurations of Kind: RANDeploymentConfiguration.
In some embodiments, the RANDeploymentConfiguration Spec CRD is as follows:
The kind keyword in this example identifies RANDeploymentConfiguration as a new CRD. The apiVersion has been set to “oran/v1alpha1.” The metadata attribute identifies the CRD name as ran-app-deployment-configuration, and provides “deployment configuration for application package” as its description.
The spec for RANDeploymentConfiguration contains applications and overrides. Applications is a list of maps. A map has one or more fields in some embodiments. These fields include name, which is a list of application types of Kind: RANApplicationType. In some embodiments, these application types include ran-xapp and ran-rapp. Overrides is a list of overrides of Kind: RANApplicationOverride. In some embodiments, RANApplicationOverride has a similar schema as RANDeploymentConfiguration with only fields that a vendor would specify as field that can be overridden.
RANApplicationType Spec specifies a list of description for various applications that need to have their lifecycles managed by this object. In some embodiments, multiple xApps and/or combination of xApp and rApp from same vendor can be described under this section. The CRD controller in some embodiments the homing based on the resource, latency and app requirements. For each of these apps, the CRD controllers on different cloud can perform lifecycle management and push towards the desired state.
Two examples of the RANApplicationType Spec CRDs are as follows. The first example is for xApps and the second one is for rApps.
Example 1:
Example 2:
The kind keyword in these example identifies RANApplicationType as a new CRD. The apiVersion has been set to “oran/v1alpha1.” The metadata attribute identifies the CRD name as ran-app-type, and provides “An instance of application type” as its description. The spec for RANApplicationType contains type, vendor, version and components. The type field indicates the application type. Supported values are “xapp” or “rapp” in some embodiments. Depending on the type, the controller could create another custom resource and do the placement on a specific K8s endpoint.
In some embodiments, components includes a list of DeployableComponents of kind: RANDeployableComponent. As shown in the examples above, each component can be specified by a name and an override. The vendor and version fields of the spec attribute of the RANApplicationType is used in some embodiments to specify the vendor and version of the RAN application.
As shown, the RANDeployableComponent CRD includes initContainers field, which is used to describe the list of the init containers that needs to be run in order to configure any initial state of the application. This is Optional field. The initContainer shares the fields such as name, image and command of the Containers attribute of this CRD.
The Containers attribute is a list of containers belonging to the pod. In some embodiments, there must be at least one container in a Pod. A container has a name, which in some embodiments is specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). In
A container also has an image specified by the image field in the CRD. The image is a docker image in this example. This name describes the repository path (docker.io/oran/appM: 1.0.0) along with registry information and tag to allow “docker pull” command to successfully download the docker image on the host where the pod is initialized. The container also includes a command attribute. In some embodiments, this attribute is an entrypoint array that is not executed within a shell. The docker image’s ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references are not expanded in some embodiments, regardless of whether the variable exists or not. They are also not updated in some embodiments.
In some embodiments, the container also includes arguments (“--ric_host”, “ric”, “--ric_port”, “5000”) to the entrypoint, while in other embodiments it does not (e.g., since the controller in these embodiments automatically configures the xApp or rApp to interact with the RIC). The docker image’s CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references are not expanded in some embodiments, regardless of whether the variable exists or not. They are also not updated.
The container has a livenessProbe attribute, which defines the periodic probe of container liveness. Container is restarted when the probe fails. This attribute cannot be updated in some embodiments. In this example, the probe attributes specifies the protocol, the path, port, initial delay of the probe and period of the probe. The container’s readinessProbe specifies periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. This attribute cannot be updated in some embodiments. Like livenessProbe, readinessProbe specifies the protocol, the path, port, initial delay of the probe and period of the probe
The resources attribute of the container attribute of the RANDeployableComponent CRD specifies the compute resources required by the container. Exemplary resources to specify in some embodiments are CPU and memory (RAM). When resource requests are specified for a container in a Pod, the scheduler uses this information to decide which node to place the Pod on. If a resource limit is specified for a Container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. The kubelet also reserves at least the request amount of that system resource specifically for that container to use. When the limits for resources are not specified, the controller can enforce an upper limit to be same as the request. In some embodiments, the resources attribute cannot be updated.
The volumeMounts attribute of the container attribute of the RANDeployableComponent CRD refers to all the files that need to mounted on the pod. All the files that need to be mounted are a reference to kind: Secret. The volumeMounts attribute has several fields, which are (1) a name field that specifies the name of the volume mount (this name should be the same name as resource kind: Secret), (2) mountPath field, which specifies the absolute path for the file in the pod, (3) subPath field, which is a reference to specific key or file name in the secret, and (4) a data field, which is a base64 value of the file content.
The affinity attribute of the container attribute of the RANDeployableComponent defines the constraints for container placement. It includes (1) a nodeAffinity that describes node affinity scheduling rules for the pod, (2) podAffinity that describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)), (3) podAntiAffinity that describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).
The ports attribute of the container attribute of the RANDeployableComponent provides a list of ports to be assigned for the container. Each port in this list is defined in terms of a name and a port number that is exposed on the pod’s IP address. The port number has to be a valid port number, 0 < x < 65536. The configuration attribute of the container attribute of the RANDeployableComponent describes generic key-value pair that can be exposed as an .ini file or as env variables in the container.
The Kubernetes cluster 3615 then processes the application and/or component descriptor CRDs to deploy the application and/or one or more of its components. In some embodiments, this deployment is facilitated by a RIC controller 3620 in the O-RAN system 3600. The RIC controller 3620 registers and receives notifications regarding for events associated with each particular CRD (including the application and component CRDs). Whenever the RIC controller receives a notification regarding a request associated with one of these CRDs (e.g., a CRUD request), the RIC controller 3620 then performs the action associated with this request.
The placer 3710 processes (at 3815) these CRDs (e.g., parses them and analyzes the parsed content) to identify a placement for the application that needs to be deployed. The placer then informs (at 3820) the deployment manager 3705 of its placement decision. The deployment manager then requests (at 3825) a cloud native deployment tool 3715 (associated with the cloud selected by the placer) to direct the selected cloud to deploy the application in the cluster selected by the placer.
The cloud native deployment tool 3715 then directs (at 3830) the selected cloud 3720 to deploy the application in the cluster selected by the placer. To so direct the cloud 3720, the cloud native deployment tool 3715 provides the application and/or component CRD descriptors through the APIs of the selected cloud. At 3835, the RIC controller of the selected cloud then receives notification regarding the deployment request that is made with respect to the custom resources, and performs the necessary operations for the deployment of the application and/or its components, and their subsequent life cycle management. After 3835, the process ends.
The Deployment Descriptor CRD describes the deployment and has the Cloud Descriptor CRD as a child CRD. The Cloud Descriptor CRD describes the attributes of a cloud to which the application should be deployed. As child CRDs, it has the Cloud Native Descriptor File CRD, and in some cases, the Component Descriptor CRD.
The App Descriptor CRD has one or more Component Descriptor CRDs as child CRDs, and describes the application contained in the App Package. Each Component Descriptor CRD describes a different component of the application. This CRD refers to a docker image, and may have one or more child RTE Descriptor CRDs. The App Descriptor CRD also has several other child CRDs that describe the accounting, performance, security, alarm and configuration model operations associated with the applications. The App Descriptor CRD, the Component Descriptor CRD, and their child CRDs are processed by RIC platform to implement applications, such as rApps and xApps.
The bus 4005 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 4000. For instance, the bus 4005 communicatively connects the processing unit(s) 4010 with the read-only memory 4030, the system memory 4025, and the permanent storage device 4035.
From these various memory units, the processing unit(s) 4010 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) 4010 may be a single processor or a multi-core processor in different embodiments.
The read-only-memory (ROM) 4030 stores static data and instructions that are needed by the processing unit(s) 4010 and other modules of the electronic system 4000. The permanent storage device 4035, on the other hand, is a read-and-write memory device. This device 4035 is a non-volatile memory unit that stores instructions and data even when the electronic system 4000 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 4035.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device 4035. Like the permanent storage device 4035, the system memory 4025 is a read-and-write memory device. However, unlike storage device 4035, the system memory 4025 is a volatile read-and-write memory, such as random-access memory. The system memory 4025 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention’s processes are stored in the system memory 4025, the permanent storage device 4035, and/or the read-only memory 4030. From these various memory units, the processing unit(s) 4010 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 4005 also connects to the input and output devices 4040 and 4045. The input devices 4040 enable the user to communicate information and select commands to the electronic system 4000. The input devices 4040 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 4045 display images generated by the electronic system 4000. The output devices 4045 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, a number of the figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process.
Also, several embodiments described above only show one hardware accelerator per host computer. However, one of ordinary skill will realize that the methodology and architecture of some embodiments can be used to provide direct, passthrough access to multiple hardware accelerators on one host computer. In addition, several embodiments described above pertain to xApp operations and the near RT RIC communications with xApps. One of ordinary skill will realize that these embodiments are equally applicable to edge applications in a telecommunication network and the near RT RIC communications with the edge applications. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
| Number | Date | Country | |
|---|---|---|---|
| 63236682 | Aug 2021 | US |
