Patent Application
20220138081
As communications network architectures and capabilities change over time, they provide new opportunities for network operators and for end users. These new opportunities may take the form of improved, and in some cases new, services that can be implemented by software applications. For example, as new generations of wireless networks are developed and made available to users, client devices can take advantage of increased bandwidth, decreased latency, and the ability to interact with the network in ways that allow the network to service the needs of an application. In this regard, previous generation of networks were application agnostic and did not adapt to an application and its network resource requirements or characteristics. However, newer and more advanced networks have the capability to be application aware, and this has implications for the types of applications and services that can be provided to users and network administrators. This may include applications being able to access services or network infrastructure functionality that was too slow or could not be taken advantage of with previous client devices and applications.
As part of providing user access to the advantages of new network architectures and capabilities, network owners, network operators, service and application developers may need to evaluate how certain applications and services perform on a specific network when it is configured according to specific parameters or under specific operating conditions. This evaluation may assist developers to optimize the performance of an application when used in conjunction with a specific client device and network configuration. A result of this evaluation or optimization process may be to identify a set of parameters for the application that provide a desired level of service performance for a user accessing the application with a specific device over the network under a set of network operating metrics or conditions. Such an evaluation may also assist a network operator or administrator to determine the impact of supporting a particular application on the network and how best to balance use of the application with quality of service (QoS) obligations the operator has to its customers.
Conventionally, the available software development kits (SDKs) and device test platforms are tailored for specific use-cases. For instance, for the case of an application that will be installed and used on a mobile phone, conventional solutions that allow these to be tested are tailored for a specific device and operating system, e.g., iOS or Android. Only limited testing is performed regarding the application's interaction with the device resources, e.g., memory, battery consumption, processing resources, etc.
However, with the advent of 5G technology, applications are not limited to those installed and accessed by a user from an end users mobile device and may also reside on a server in the network cloud. These cloud-based applications may be consumer-oriented applications such as those found on consumer phones, and (or instead) may be applications for other devices and purposes. These might include, for example, medical devices in the healthcare field, IoT controllers, and applications for specialized verticals such as transportation, enterprise, entertainment, financial, education, or agriculture.
These cloud-based applications may include specialized applications for use in managing a network infrastructure by accessing, monitoring, and configuring network elements and network functions. As another example, a cloud-based application might be used to configure or monitor off-the-shelf hardware. For example, Raspberry Pi, Arduino etc., are typically referred to as COTS (Commercial Off The Shelf) hardware and may be used for embedded applications i.e., an application running on a general-purpose processor on these boards.
There is a need for all interested parties (application developers, service developers, service owners, network owners, and network operators/administrators) to be able to orchestrate or evaluate the performance of an application in a situation as dose as possible to what would be expected in an actual network prior to deploying the application. This is desirable to properly evaluate the performance of an application and the demands placed by the application on a network prior to a full deployment, as those may impact the end-user experience or the allocation of network resources to the application. Post evaluation, network administrators, operators and/or owners can then orchestrate the integration of the service or application into the actual network configuration.
Further, conventional application testing approaches are not intended for, and in most cases, not capable of being used to evaluate the performance of an application within a specific network configuration (referred to as a network “slice” and which may define a specific level of speed, latency, reliability, and security), much less to assist an application developer to optimize the performance of their application for one or more network configurations or allow a network owner or administrator to successfully orchestrate the integration of the application into an actual network.
Thus, systems and methods are needed for more efficiently and effectively enabling the testing and evaluation of the performance of an application when used in a specific network configuration. Embodiments of the disclosure are directed toward solving these and other problems individually and collectively.
The terms “invention,” “the invention,” “this invention,” “the present invention,” “the present disclosure,” or “the disclosure” as used herein are intended to refer broadly to all the subject matter described in this document, the drawings or figures, and to the claims. Statements containing these terms should be understood not to limit the subject matter described herein or to limit the meaning or scope of the claims. Embodiments covered by this disclosure are defined by the claims and not by this summary. This summary is a high-level overview of various aspects of the disclosure and introduces some of the concepts that are further described in the Detailed Description section below. This summary is not intended to identify key, essential or required features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, to any or all figures or drawings, and to each claim.
In some embodiments, the systems, apparatuses, and methods described herein are directed to the testing and evaluation of an application's performance when the application is used with a specific network architecture and configuration. In some embodiments, a testing and evaluation platform recommends, generates, and executes end-to-end network testcases based on the type and characteristics of the application, and one or more network configuration parameters. In some embodiments, the platform may provide performance-specific test measurements for an application gathered from measured network parameters (such as KPIs). The parameters may be collected during a simulation or emulation of the application during its use over the network architecture and with a specific configuration of network parameters. As an example, the described test platform and associated processing methods may provide test metrics with respect to network connection, bandwidth, and latency performance of an application over the configured network. In some embodiments, the platform may provide orchestration for integration of the application into a network for testing or actual deployment purposes.
This information may provide an application developer with a deeper understanding of an application's interaction with the network, and its expected performance under one or more of ideal, best, expected (nominal or standard), or even degraded network operating conditions. This can assist the developer to make performance enhancing changes to one or more of the application's data processing or workflow, resource access and usage, network traffic, enabled or disabled options, features, or default features or operations, among other aspects.
As a non-limiting example, a video streaming application may choose to disable video compression of a 720p or 1080p video while transmitting over a 5G network to provide a better user experience. The impact of this change on the application can be only tested over a live 5G network. Instead of driving around to access a 5G network, a developer may choose to test over the test platform described herein that provides access to a 5G network and to meaningful measurements made in the network. The described test platform can provide a developer with a measurement of the bandwidth consumed over the network so that it may be compared to the available bandwidth in the network. An application developer can then choose to stream a higher fidelity video e.g., HD or 4K based on the bandwidth consumption measured over the live network and with awareness of the direct impact of disabling the compression function.
In some embodiments, the systems and methods described herein provide application integration, application testing, and network performance services through a SaaS or multi-tenant platform. The platform provides access to multiple users, each with a separate account and associated data storage. Each user account may correspond to an application developer, group of developers, network owner, network administrator, or business entity, for example. Each account may access one or more services, an example of which are instantiated in their account and which implement one or more of the methods or functions described.
In one embodiment, the disclosure is directed to a method for enabling the testing, monitoring, and evaluation of the performance of an application or service on a configuration of a telecommunications network. In one embodiment, the method may include the following steps, stages, functions, processes, or operations:
In one embodiment, the disclosure is directed to a system for the testing, monitoring, and evaluation of the performance of an application on a configuration or configurations of a telecommunications network. The system may include a set of computer-executable instructions and an electronic processor or processors. When executed by the processor or processors, the instructions cause the processor or processors (or a device of which they are part) to perform a set of operations that implement an embodiment of the disclosed method or methods.
In one embodiment, the disclosure is directed to a set of computer-executable instructions, wherein when the set of instructions are executed by an electronic processor or processors, the processor or processors (or a device of which they are part) performs a set of operations that implement an embodiment of the disclosed method or methods.
Other objects and advantages of the systems and methods described will be apparent to one of ordinary skill in the art upon review of the detailed description and the included figures. Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.
Embodiments of the present disclosure will be described with reference to the drawings, in which:
The subject matter of embodiments of the present disclosure is described herein with specificity to meet statutory requirements, but this description is not intended to limit the scope of the claims. The claimed subject matter may be embodied in other ways, may include different elements or steps, and may be used in conjunction with other existing or later developed technologies. This description should not be interpreted as implying any required order or arrangement among or between various steps or elements except when the order of individual steps or arrangement of elements is explicitly noted as being required.
Embodiments of the disclosure will be described more fully herein with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, exemplary embodiments by which the disclosure may be practiced. The disclosure may, however, be embodied in different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy the statutory requirements and convey the scope of the disclosure to those skilled in the art.
Among other things, the present disclosure may be embodied in whole or in part as a system, as one or more methods, or as one or more devices. Embodiments of the disclosure may take the form of a hardware implemented embodiment, a software implemented embodiment, or an embodiment combining software and hardware aspects. For example, in some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented by one or more suitable processing elements (such as a processor, microprocessor, CPU, GPU, TPU, controller, etc.) that is part of a client device, server, network element, remote platform (such as a SaaS platform), an “in the cloud” service, or other form of computing or data processing system, device, or platform.
The processing element or elements may be programmed with a set of executable instructions (e.g., software instructions), where the instructions may be stored on (or in) one or more suitable non-transitory data storage elements. In some embodiments, the set of instructions may be conveyed to a user through a transfer of instructions or an application that executes a set of instructions (such as over a network, e.g., the Internet). In some embodiments, a set of instructions or an application may be utilized by an end-user through access to a SaaS platform or a service provided through such a platform.
In some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented by a specialized form of hardware, such as a programmable gate array, application specific integrated circuit (ASIC), or the like. Note that an embodiment of the inventive methods may be implemented in the form of an application, a sub-routine that is part of a larger application, a “plug-in”, an extension to the functionality of a data processing system or platform, or other suitable form. The following detailed description is, therefore, not to be taken in a limiting sense.
As mentioned, in some embodiments, the systems and methods described herein provide application integration, application testing, and network performance services through a SaaS or multi-tenant platform. The platform provides access to multiple users, each with a separate account and associated data storage. Each user account may correspond to an application developer, group of developers, network administrator, or business entity, for example. Each account may access one or more services, an example of which are instantiated in their account and which implement one or more of the methods or functions described.
Embodiments of the disclosure are directed to systems, apparatuses, and methods for the testing and evaluation of an application's performance when the application is integrated with a specific network architecture, configuration, and service. This is a more complex endeavor than it might appear at first, as there are several interrelated factors:
To properly address these factors and the various combinations, in addition to providing an Application Performance Evaluation and Application Integration platform or system, some embodiments provide the following functions or capabilities:
As mentioned, there are three primary classifications or categories of applications that may be tested:
This class of applications use the network as a content delivery pipeline. These applications consume bandwidth on the network and may or may not be latency sensitive. They may require a quality of experience (QoE) or quality of service (QoS) evaluation over a network based on the bandwidth consumption and latency behavior of the application in relation to the bandwidth and latency capacity of the network. Examples of this category of application include YouTube, Netflix, Facebook, Messenger, and Whatsapp.
This class of applications interact with the network to request network resources. The network resources may be used in different configurations to create multiple service classifications for an application, and with that different levels of quality of experience for a user interacting with the application. The application test services are performed on a slice type requested from the network, and it is determined if that allows the application to operate properly based on the network resources granted. In some embodiments, native 5G applications may request a network slice in terms of one or more of the following parameters: eMBB (enhanced Massive Broadband), mMTC (massive Machine Type Connect), URLLC (Ultra Reliable Low Latency Communication), or a combination of them: eMBB+URLLC or mMTC+URLLC. V2X applications and XR (Mixed Reality, Virtual Reality) applications are examples of native 5G applications.
This class of applications are built for purposes of Network Management, Network Operations, Network Security, and related functions. The 5G networks are for the first time open to third-party applications that may be added to the Network appliances. This further illustrates how third-party applications benefit from access to a network environment where the applications can be tested before production and deployment. Examples of these applications are machine learning models that organize radio resources based on enrichment data, where enrichment data is data received from outside the network, e.g., from external resources such as websites, internet directories, data collection sites, event management sites, etc.
Note that several 5G network bandwidth classifications may exist. For instance:
Since a 5G network is application aware and applications can request network resources, application requests for a network slice (resource) should be tested with the network, along with the network slice grant. Note that testing over Wi-Fi does not suffice, because Wi-Fi does not provide the speeds available with 5G and does not provide mobility management functions used to guarantee a level of experience for a user of an application.
Based on the network configuration required to auto-run the testcases, the application is integrated 209 into the respective network. Before the application is integrated into the network, application undergoes a security analysis 206 to determine the security risk of installing the application in the desired network appliance. An application may be installed on a user equipment such as smart phone, a special user provided emulator connected to the network over a wireless interface, commercial off the shelf hardware such as a Raspberry Pi or Arduino, an edge compute server, or a network appliance.
The required tools 211 in the network are enabled for the testcases to be executed 212. The logs are collected from the tools 213 and stored in a database 214 for further calculation and analysis. Visualization graphs 217 are built using the analysis 216 and calculated data. These results 218 and logs are returned to the user dashboard 207 to show the results of testcase execution and metrics collected and analysis performed.
The front-end 310 is implemented as a SaaS platform in the cloud providing each access to users through account signups and logins. The middleware 320 is implemented in a network environment on servers providing functions of testcase generation, automation, and orchestration 311 along with network orchestration 313 (MANO) to setup the required network configuration as required by the auto-generated testcases. The back-end 330 is implemented as a live 4G and/or 5G network that is slice capable with complete setup of User Equipment, Radio, Routing equipment and 5G and 4G core services, along with edge cloud servers available to host mobile edge applications.
The application modules and/or sub-modules may include any suitable computer-executable code or set of instructions (e.g., as would be executed by a suitably programmed processor, microprocessor, or CPU), such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer-executable code. Alternatively, or in addition, the programming language may be an interpreted programming language such as a scripting language.
Each application module or sub-module may correspond to a particular function, method, process, or operation that is implemented by execution of the instructions contained in the module or sub-module. Such function, method, process, or operation may include those used to implement one or more aspects, techniques, components, capabilities, steps, or stages of the described system and methods. In some embodiments, a subset of the computer-executable instructions contained in one module may be implemented by a processor in a first apparatus and a second and different subset of the instructions may be implemented by a processor in a second and different apparatus. This may happen, for example, where a process or function is implemented by steps that occur in both a client device and a remote server or platform.
A module may contain computer-executable instructions that are executed by a processor contained in more than one of a server, client device, network element, system, platform or other component. Thus, in some embodiments, a plurality of electronic processors, with each being part of a separate device, server, platform, or system may be responsible for executing all or a portion of the instructions contained in a specific module. Thus, although
The function, method, process, or operation performed by the execution of instructions contained in a module may include those used to implement one or more aspects of the disclosed system and methods, such as for:
As shown in
As an example, Obtain Data Characterizing Application to be Tested Module 406 may contain instructions that when executed perform a process to obtain from an application developer certain information used to configure and execute the Application Performance Evaluation and Application Integration processes. This may be done through a series of questions that are logically arranged to obtain the information based on answers to questions or data provided.
Generate Application Profile—Optimize Profile Using Trained Learning Process Module 408 may contain instructions that when executed perform a process to generate a profile of the application based on the developer's inputs and if needed, optimize or revise that profile using a trained learning process or model.
Generate Test Case Profile Using Trained Learning Process—Encode and Provide to Test Case Generator Module 410 may contain instructions that when executed perform a process to generate a test case profile based on the application profile using a trained learning process or model, encode the test case profile and provide the encoded profile to a test case generator function.
Generate Test Case(s) Module 411 may contain instructions that when executed perform a process to generate one or more test cases for the application that will determine its operation and performance in a specified network configuration.
Determine Optimal (or Desired) Network Test Bed for Test Case(s) Module 412 may contain instructions that when executed perform a process to determine a network configuration for execution of the test case or cases. In some embodiments, this may be the optimal configuration, while in others it may be a sub-optimal configuration, such as one intended to evaluate the performance of the application during a network connectivity or bandwidth problem.
Execute Test Case(s) Module 414 may contain instructions that when executed perform a process to execute the one or more test cases within the specified network test bed and configuration.
Collect Test Case Log Data, Generate Result Profile, Map to Performance Profile Module 415 may contain instructions that when executed perform a process to collect log or other data produced by the application and testing processes during testing of the application, process that data to produce a test result profile, map the test results to the application performance, and make that information and data available to the developer in one or more forms (such as the displays and graphs described herein or various tables or metrics).
In some embodiments, the functionality and services provided by the system and methods described herein may be made available to multiple users by accessing an account maintained by a server or service platform. Such a server or service platform may be termed a form of Software-as-a-Service (SaaS).
In some embodiments, the application testing and evaluation system or services described herein may be implemented as micro-services, processes, workflows, or functions performed in response to the submission of an application to be tested. The micro-services, processes, workflows, or functions may be performed by a server, data processing element, platform, or system. In some embodiments, the application testing and evaluation services may be provided by a service platform located “in the cloud”. In such embodiments, the platform may be accessible through APIs and SDKs. The functions, processes and capabilities described herein and with reference to the Figures may be provided as micro-services within the platform. The interfaces to the micro-services may be defined by REST and GraphQL endpoints. An administrative console may allow users or an administrator to securely access the underlying request and response data, manage accounts and access, and in some cases, modify the processing workflow or configuration.
In accordance with the advantages of an application service provider (ASP) hosted business service system (such as a multi-tenant data processing platform), users of the services described herein may comprise individuals, businesses, stores, organizations, etc. A user may access the application testing and evaluation services using any suitable client, including but not limited to desktop computers, laptop computers, tablet computers, scanners, smartphones, etc. In general, any client device having access to the Internet may be used to provide an application to the platform for processing. Users interface with the service platform across the Internet 512 or another suitable communications network or combination of networks. Examples of suitable client devices include desktop computers 503, smartphones 504, tablet computers 505, or laptop computers 506.
Application Performance Evaluation and Application Integration system 510, which may be hosted by a third party, may include a set of application evaluation and integration services 512 and a web interface server 514, coupled as shown in
In some embodiments, the set of services available to a user may include one or more that perform the functions and methods described herein for application testing, evaluation, and reporting of application performance results. As discussed, the functions or processing workflows provided using these services may be used to perform one or more of the following:
As examples, in some embodiments, the set of application testing, evaluation, and reporting functions, operations or services made available through the platform or system 510 may include:
Based on the application profile, the platform generates a testcase profile, a performance profile, and a results profile. These profiles are used to generate the application results dashboard. The dashboard shows the status of the testcase execution based on the testcase profile. The performance profile is used to create 3 separate evaluations for (1) viable performance, (2) plug and play performance, and (3) predictable performance. The results profile lays out the metrics collected for the testcase profile and performance profile.
The test results returned from calculated metrics and analysis by the backend are used by the middleware to generate visualization graphs. Metrics, logs, and graphs are packaged by the middleware and sent to the front end for display on the dashboard based on the results profile, performance profile and testcase profile generated by the front-end platform.
The back end also contains a component which interacts with the testcase orchestrator in the middleware, referred to as Testcase RPA (Robotic Process Architecture) 701. The RPA is an agent responsible for network command execution over a live network. It also contains the tools, for example, network probes 702 and log collection connectors that collect the data into a time stamped database for metric calculation and analysis. This calculated metric and analysis is passed to the middleware for visualization generation by middleware. Each network that is added to the test bed contains a back-end platform which hosts the tools and components for a successful application performance test execution.
The cloud-based application performance evaluation and integration platform described herein provides access to application developers and test assurance companies to test an application over a live network. The testing furnishes metrics and KPIs that enable application developers to correlate application features and behavior to performance characteristics observed in the network for the application under test. The deep packet inspection tools that are used to collect intelligence from the network about the application interaction are not provided in conventional live networks and such networks do not publish their performance KPIs for use externally.
These KPIs can help applications confirm that the network service level agreement (SLA) setup by requesting the QoS is able to be maintained by the network and the QoS is not deteriorated under ideal conditions of the network. Note that the QoS will deteriorate under non-ideal conditions of the network, such as when the network experiences higher traffic leading to congestion or when there is a failure in a network component causing network disruption.
The disclosed system/platform allows an application to test under non-ideal conditions to confirm how the QoS on the application behaves (such as by deteriorating) and recovers once the network recovers from congestion or disruption. These results are captured by the predictable performance assurance profile for testcases run under this category of performance assurance.
In an example use of the systems and methods described an application under test is recorded as its functionality is executed and an active co-relation to application performance over the network is provided. In one embodiment, a video in a frame is provided alongside the graphs generated from the testing process outputs. As a user moves a cursor across the graph, the video plays back to the same time as the specific time on the graph. Further, the test script log may be superimposed for a specific point on the graph. Graphs may be placed next to each other with a facility to mark a specific graph with the same marker showing up on other graphs for easier correlation.
In this sense, the testing and evaluation processes determine how a network characteristic might impact (or be impacted by) the use of a feature in an application (and in what way) so that measuring a network parameter and its changes during use of the application can provide an indication of how an application will perform or can be used with a given network configuration.
In some embodiments, network metrics are collected using a client embedded in a network with the ability to discover a network topology, decide where best to insert probes, activate the probes for deep packet inspection to acquire operational metrics in real-time and thereby to identify network performance during testing of an application.
Network topology discovery is a process that determines the interfaces on the network (for example, as illustrated in
In some embodiments, the specific network metrics furnished for applications under test may comprise one or more of:
The ability to capture KPIs and metrics of these types and at this level is not available on standard consumer networks (i.e., actual operating 5G networks available for use by consumers) for several reasons. To extract this information, consumer networks would need to support deep packet inspection capabilities and the return of the relevant parameters back to the user. This capability is not supported or provided by the widely available consumer networks. In addition, such networks typically do not provide the types (or classifications) of access desired by application developers to enable them to test their applications over a wide variety of the possible 5G scenarios. These scenarios may include testing an application under ideal conditions, testing the application while disrupting network connectivity and checking how the application recovers from it, and testing the application under congested network conditions.
The systems and methods described herein are motivated, at least in part, by a need for interested parties (e.g., application designers, developers, network administrators, network owners, service developers and application testers) to be able to test or evaluate the performance of an application in a realistic network configuration prior to launch and deployment of the application. With the advent of 5G networks, this is rarely possible and may result in a costly development and roll-out process, only to require later patches or modifications to an application. This is even more likely given the nature of 5G network variabilities, which may depend on the spectrum band the network is deployed on.
In contrast, previous 4G LTE networks only existed in low-band spectrum which resulted in a similar behavior from network to network. Moreover, most developers tested over Wi-Fi, which provided comparable results to those obtained over the actual network. However, given the Gbps data speeds and low latencies available in sub-mm wavelength bands over 5G networks, Wi-Fi does not provide the appropriate network characteristics to test high bandwidth, low latency, or machine-to-machine applications that require mobility management.
Embodiments provide a test or evaluation platform that can be used to simulate or emulate a network having a specific configuration so that the performance of both an application and network can be monitored during use of the application. This may result in changes to an application's feature set, to the implementation of a feature, to network management resource request rules for the application, or to another application or network related modification.
Embodiments enable an application developer to observe the impact of performing an activity over a 5G network, that is no longer considered a metered resource by a device (as some prior 4G and other networks may have been treated). These new networks allow or more effectively enable many desirable activities by an application, such as the pre-fetch of high bandwidth content without waiting to connect to a Wi-Fi network, activating or deactivating real time codecs to allow for higher quality content playback, and re-designing or de-activating compression algorithms which are not required over a high-speed network.
The benefits of 5G networks allow for different services and qualities of experience for users, performing activities that have not been tried previously over a regular Wi-Fi or LTE network due to a lack of sufficient network resources being available. Typically, these network resources are reduced or rationed across users to provide equal quality of service to all users on LTE and predecessor networks, for example by playing 4K or 8K content over the network that was only supported over optical broadband connections to a home. In contrast, and for the first time, 5G networks allow for service classification to different sets of users based on the quality of experience (QoE) they want to be provided with and are willing to be charged accordingly.
In some embodiments, the application testing system and methods described herein provide access to a private test network through a cloud-based server/platform that includes the ability to deploy deep packet inspection techniques to measure and evaluate the network interaction with an application.
Historically, the “best” application experience is usually available over Wi-Fi and most apps test over a local enterprise Wi-Fi. However, this type of testing does not assist in evaluating the ability to meet a guaranteed Service Level Agreement (SLA). With speeds up to 1 GBps available, meeting the terms of an SLA has not been a concern on Wi-Fi. Additionally, Wi-Fi is based on accessing a shared resource using contention-based protocols with collision detection CSMA-CD (Carrier Sense Multiple Access with Collision Detection) that allows for several devices accessing the medium and if there is a contention, the devices back off for an arbitrary amount of time to resolve the contention. Wireless networks do not employ similar contention-based protocols. Additionally, a problem with this form of testing is that Wi-Fi does not offer mobility management, which is highly desirable and almost required for much of the IoT applications and rich user experience applications of interest to users.
With the arrival of 5G networks, the best application experience will be available over a wireless network rather than an enterprise Wi-Fi network. This creates a strong motivation, and realistically a need, to be able to test an application over a network. Also, a network may provide a guaranteed SLA and allows the implementation of a Service Oriented Architecture (SOA) that exposes network APIs to allow an application to request network resources. Because of this capability, an application needs to understand how much to ask for and whether those grants of network resources help them to create differentiated service levels for their applications and users.
In one embodiment, a method for the testing and evaluation of an application's performance when the application is used with a specific network architecture and configuration may comprise the following steps, stages, operations, processes, or functions, as described in the following sections.
The Application Profile Model (APM) is used to develop or create a profile for an application to be tested/evaluated:
Based on the developed profile for the application and the defined network environment(s), generate a set of test cases for the application;
Viable Performance Assurance
This performance measure provides a preliminary assessment of an application's performance on a network is based on a “standard” deployment (i.e., an initially assumed or default deployment or configuration) of 5G technology. This is the performance that an application expects from the network to meet throughput and latency requirements and is used to establish a standard Quality of Experience for the application. An example deployment may have the following parameters or metrics:
Plug and Play Performance Assurance
This performance measure corresponds to the performance that the application guarantees for smooth interoperability over a variety of networks worldwide. This is to establish the interoperable Quality of Experience for the application. Vertical represents an industry specific application and these measures refer to Layer 1 performance of the application (referring to layer 1 of the OSI layer diagram). These metrics may include: Application device CPU performance, Application device tasks, Application device battery consumption, and Radio connectivity (L1) to application device performance.
Predictable Performance Assurance
This deployment definition may be based on one or more 5G Service Slice Types1, for example:
From a business model perspective, each network slice is administrated by a mobile virtual network operator (MVNO). The infrastructure provider (the owner of the telecommunication infrastructure) leases its physical resources to the MVNOs that share the underlying physical network. According to the availability of the assigned resources, a MVNO can autonomously deploy multiple network slices that are customized to the various applications provided to its own users.
This is an indication of the performance that the application needs to satisfy for a variety of network conditions (congestion, disruption) and network configurations (network slices). In one sense, this is to benchmark the minimum and maximum Quality of Experience and reliability for the application.
In some embodiments, Reliability is measured in terms of Continuity, Availability & Recoverability. Continuity primarily tests for application reliability for short duration network failures. Recoverability primarily tests for application reliability in terms of time to recover from failures. Availability primarily tests for application reliability in terms of recovery after multiple failures of different durations. The dashboards displayed and arrangement of testcases under those dashboards are determined from the performance profile. The performance profile also determines which KPIs and metrics are mapped to each dashboard.
For each executed test case in the set of test cases, collect information/data on the network and/or application performance during the execution of the test case. The collected information/data may comprise:
Next, process the collected information/data to represent it in the form of a dashboard of metrics, graphs, and illustrations;
Further, in some embodiments, the system may generate a recommendation to an application developer or network administrator regarding a way to improve the performance of the application on the specific network environment.
In some embodiments, a part of the dashboard is a set of one or more visualization graphs 1503 for the network throughput measured while application under test is running over the network. Overlaid next to the graphs (not shown) is a video of the application executing so that network administrators and application developers can correlate the execution of the application with a specific action of the application produced the specific network effect as observed in the Network Throughput graph. A static analysis of the application code is also available to co-relate the specific line of code that may be under execution to produce the network effect depicted on the graph.
In some embodiments, a part of the dashboard is a set of one or more visualization graphs for the application device performance measured while application under test is running on the device. The device could be a srnartphone, specialized hardware, COTS hardware, Edge Server. Overlaid next to the graphs (not shown) is a video run of the application so that network administrators and application developers can co-relate the execution of the application and which specific action of the application produced the specific device performance or network connectivity performance as observed in the application device task count 1602 or the application device CPU performance 1603. A static analysis of the application code is also available to co-relate the specific line of code that may be under execution to produce the device performance effect depicted on the graph.
This category of performance assurance depicts the plug and play performance of an application. In the sample application under test, the measurements captured are application device performance on which the application under test is executing. It also shows the radio connectivity performance using the radio parameters to confirm the quality of wireless signal over which the application is transmitted by the application device. These parameters namely RSRQ 1605 and 1606, RSRP, SINR 1604 and CQI indicate the quality of layer 1 (transmission medium) over which the application is interacting with the network. The quality of this medium is directly co-relational to the QoE and QoS of the application under test.
Note that the components and system elements described, which may be identified as Front End, Middleware, and Back End components and processes are typically implemented using one or more cloud infrastructures and dedicated physical servers on lab premises in one or more geographic locations. As such, in some embodiments, the system represents an example of a distributed service-oriented architecture (SOA) design.
Among other features or functions, embodiments of the systems and methods may provide one or more of the following services or features:
Conventionally, an obstacle to designing and implementing an effective application testing platform has been enabling the platform to accept a wide range of applications for evaluation. In this regard, conventional evaluation or testing services accept smart phone applications while the rest of the network is virtualized (i.e., it is a simulated network).
This limitation imposed by conventional approaches can reduce the value of a platform or system as applications can range from consumer-oriented applications executed on different consumer devices, to more specialized applications executing on specific hardware, and may include distributed applications accessed from a remote server. Further, in some cases, it may be desirable to test or evaluate content with regards to its ability to be transferred over a network and used by an application.
As suggested, locations for application installation have become more numerous because of the nature of 5G network architecture and a service-based network architecture. The disclosed platform and associated services are designed to permit evaluation and testing of many types of applications, which adds to the complexity of the platform design. Some of these complexities are addressed by creating application profiles for the various types of applications. Further, the live network used in embodiments of the systems and methods disclosed herein also provide access to edge compute servers to allow applications to install on the edge, as well as allowing applications to install in the network appliances.
The system and platform described herein overcome these limitations and provides an effective and scalable set of application testing and evaluation services. In some embodiments, the testing processes performed by the system and platform may include one or more of end user device or hardware testing, network testing, monitoring an application or applications as they interact with the network, usage of network bandwidth, round trip times for latency, and a measure of the overall end-to-end quality of a user's experience with the application.
In some embodiments, to accommodate a broad range of applications and content, the platform may perform one or more of the following functions, processes, or operations:
As part of developing the described application testing services and features, a capability to autonomously perform application testing was developed. This capability includes several functions or features, including but not limited to:
Network probes were developed for deep packet inspection and active raw data reading. Connectors continuously move read data from probes to a central database over a network bus. Recorders write moved data to a central time stamped database. Correlation and measurement tools termed metric calculators were written to perform active calculation on the recorded database values. Code analyzer tools were written for static code analysis against the time stamped database values;
In some embodiments, the application testing and evaluation approach described herein and implemented by the system or platform includes monitoring both network and application performance. This monitoring may be a process that generates or acquires metrics for various components/layers of a 5G network. The application monitoring metrics may be collected as part of an application profile. In some cases, an application developer may provide an approximate application profile to the testing and evaluation system. The system monitors the application profile metrics and additional aspects of the network and application performance. This provides an opportunity for an application designer to obtain a more complete understanding of an application's behavior and usage of network resources under a variety of conditions.
The wireless technologies and networks, including network equipment, follow Industry standards developed by International Telecommunication Union (ITU), European Telecommunication Standards Institute (ETSI) and 3rd Generation Partnership Program (3GPP). Although testing of network equipment and technologies has been standardized using recommendations published by these bodies, testing of applications has conventionally been an ad-hoc endeavor lacking structure or formal requirements.
To allow for the test platform to adapt to the changing needs of technology and standards development, the test cases allow for modular inclusion of new recommendations received from standards bodies and organizations. As an example, as new KPIs and further adaptation to more advanced technologies in the future (e.g., 6G) occur, these can be incorporated by adding test components specific to 6G standards in a modular fashion, while continuing to utilize the base process automation architecture to construct testcases using modular testing components.
A design goal of the disclosed test system architecture is to modularize the construction of its front-end, middleware, and back-end components and processes. This allows those components and processes to be implemented as micro-services and enables them to adapt and change, while maintaining the user (for example, an application developer, network operator, or network administrator) experience of interacting with the platform. Specifically, the platform defines testing categories which are network centric but are network technology agnostic. The testing categories are defined and automatically selected with reference to the type of application being tested. This approach is important to provide a standardized benchmarking for all applications irrespective of the type of network or network configuration they are being tested on.
In some embodiments, each Network slice may be associated with a specific service level requirement (SLR). For example:
Data generated by the execution of relevant test cases, subject to an architecture deployment (cloud core implementation vs. on premise core implementation) and service slice type, are gathered, analyzed, and summarized for users of each vertical (where vertical is typically an industry specific application). This helps to characterize the behavior of a 5G-compatible application and end-user device, under a variety of internal and external operating conditions.
The platform and functionality described herein reduce the effort required for testing 5G infrastructure and components and evaluating the performance of an application. By simplifying the testing operations and providing a Continuous Integration (CI) pipeline as a built-in function, the platform can ensure a stable performance.
For example, a network administrator can use the platform to bring new applications into production networks and/or update existing applications with recurring releases, thus providing a continuous integration functionality for the production network with continuous updates from applications developers. Similarly, an application developer can test and certify the application on a test network through the platform and then network administrators can bring in the application and test its performance on a specific production network.
The platform serves as an automation platform for validating and verifying the entire 5G system, from the individual components to the E2E service. This is accomplished, at least in part, by the ability to abstract the complexity involved in testing the various layers ranging from Conformance to Security and from Performance to QoE.
In some embodiments, the platform includes test and measurement tools useful for validating and verifying 5G systems. These tools are used for both standard network test cases as well as custom test cases for vertical application testing. As other types of tests are developed, they can be made available through the platform.
In some embodiments, the available tools or testing functionality may include, but are not limited to or required to include:
In some embodiments, the systems and methods described herein include or implement one or more of the following decision processes, routines, functions, operations, or data processing workflows for the indicative platform function or feature:
For Application Developers, providing early-stage testing of their use cases over a standards-based full-chain 5G system or emulation, and following a systematic approach, enables a range of vertical industries to make timely and well-informed business decisions regarding launching their services and offering guaranteed service performance levels. This will ensure greater end user satisfaction and fewer network associated problems, and therefore a higher likelihood of business success.
The systems and methods described provide End-to-End (E2E), NFV characterizations, along with performance evaluation in a 5G heterogeneous infrastructure (including generalized Virtualization, Network Slicing and Edge/Fog Computing). The systems and methods include Test and Measurements (T&M) procedures, tools, and methodologies (i.e., testing, monitoring, analytics, and diagnostics) to ensure a robust, carrier-grade geographically diverse 5G Test Infrastructure.
Considering a vertical innovation lifecycle, verticals planning to leverage 5G as a key enabler in their development process are expected to face the challenge of developing and validating new solutions. These may include:
1. Verticals addressing a basic business need for their operations and/or customers which is dependent on and sensitive to the underlying communications network's performance, can utilize the systems and methods described to test and evaluate their applications and assumptions prior to roll-out in a network. The expectations on their applications for meeting extreme network reliability, sustained high throughput levels, or close to real-time communication services (to mention examples of potential requirements) need to be carefully assessed versus the 5G technology performance benchmarks to provide a viable/reasonably achievable assurance of performance;
2. 5G applications should behave properly within their specific and expected performance levels, and according to prediction models, thus confirming that well-defined objectives of an SLA are attainable and “guaranteed” by the underlying 5G network, and are satisfied for a variety of application scenarios and 5G network configurations and conditions (to generate a measure of the predictable performance of an application under realistic network operating conditions); and
3. Customers (developers) that expect to scale and reach a global market expect smooth interoperability and guaranteed performance levels with a variety of commercial 5G networks worldwide that their applications will be deployed upon (sometimes referred to as plug-and-play performance assurance).
The concurrence and convergence of fast-paced innovation at a variety of verticals with the development and roll-out of 5G by the global communications ecosystem brings new opportunities, but also poses additional risks and challenges, especially for pioneering initiatives. 5G literature lists 5G KPIs as associated with values for maximum theoretically achievable performance. However, there are several 5G Service Slice Types, such as eMBB, URLLC, and mMTC, that may condition or modify a specific set of 5G KPIs associated with an application.
Unfortunately, a commercially deployed 5G network is not the type best suited to providing an environment that verticals can utilize for completing the various stages of application development and testing. In most cases, the insights regarding application behavior and performance needed by a vertical to complete their innovation cycle can best be provided by an experimentation facility that provides them with the tools and processes to carry out testing and measurement activities, and to explore the impact of variations in application parameters, network operating parameters, and KPIs on application behavior and the experience of end users.
The systems and methods described provide access to this type of experimental network lab through a platform. Further, for understanding the desired Quality of Service/Experience (QoS/QoE) for an application over a 5G network, it is important to understand a developer's needs and network connectivity expectations, and to translate them into suitable network configurations and the selection of appropriate technological features. The use of the described systems and methods can provide guidance or recommendations for the provisioning of an optimum 5G slice selection of suitable SW and HW components in the Core, Transport and Radio network per vertical industry, with guaranteed performance metrics. This will result in better application behavior and end user satisfaction.
As an example of how the test results may be used:
This benchmarking is further analyzed against the specific 5G network slice type. Applications can be tested for specific network slice type or can be tested for all network slice types:
The disclosure includes the following clauses and embodiments:
1. A method for evaluating the performance of an application when used with a network configuration, comprising:
obtaining data characterizing an application to be evaluated;
generating one or more test cases for the application based on the data characterizing the application;
determining a network configuration for each test case;
executing each test case in a live network having the specified network configuration;
obtaining data from the execution of the test case using a deep packet inspection probe inserted into the live network;
correlating the obtained data from the execution of the test case to a performance profile for the application; and
providing the performance profile to a developer of the application.
2. The method of clause 1, wherein the data characterizing the application to be evaluated further comprises one or more of a device type, an operating system for the device, a requested network slice type, and whether the application is expected to be bursty, continuous, bandwidth intensive, or latency sensitive with regards to network usage.
3. The method of clause 1, further comprising generating a test case profile from the data characterizing the application, and the test case profile is generated using a trained model.
4. The method of clause 1, wherein determining a network configuration for each test case further comprises determining one or more of bandwidth, network protocol, frequency band, and network slice for the test case.
5. The method of clause 1, wherein the live network comprises user equipment, a radio, routing equipment, and one or both of a 5G and 4G core service.
6. The method of clause 1, wherein the deep packet inspection probe is a plurality of such probes, with each probe inserted at an interface in the live network.
7. The method of clause 1, wherein the performance profile provided to the developer is a graph or table illustrating a consumption of a network resource by the application during the test case.
8. The method of clause 7, wherein the performance profile is generated under one of a set of possible operating conditions.
9. A system for evaluating the performance of an application when used with a network configuration, comprising:
one or more electronic processors configured to execute a set of computer-executable instructions; and
the set of computer-executable instructions, wherein when executed, the instructions cause the one or more electronic processors to
10. The system of clause 9, wherein the data characterizing the application to be evaluated further comprises one or more of a device type, an operating system for the device, a requested network slice type, and whether the application is expected to be bursty, continuous, bandwidth intensive, or latency sensitive with regards to network usage.
11. The system of clause 9, wherein the instructions further cause the one or more electronic processors to generate a test case profile from the data characterizing the application, and the test case profile is generated using a trained model.
12. The system of clause 9, wherein determining a network configuration for each test case further comprises determining one or more of bandwidth, network protocol, frequency band, and network slice for the test case.
13. The system of clause 9, wherein the live network comprises user equipment, a radio, routing equipment, and one or both of a 5G and 4G core service.
14. The system of clause 9, wherein the deep packet inspection probe is a plurality of such probes, with each probe inserted at an interface in the live network.
15. The system of clause 9, wherein the performance profile provided to the developer is a graph or table illustrating a consumption of a network resource by the application during the test case.
16. A set of computer-executable instructions that when executed by one or more programmed electronic processors, cause the processors to evaluate the performance of an application when used with a network configuration by:
obtaining data characterizing an application to be evaluated;
generating one or more test cases for the application based on the data characterizing the application;
determining a network configuration for each test case;
executing each test case in a live network having the specified network configuration;
obtaining data from the execution of the test case using a deep packet inspection probe inserted into the live network;
correlating the obtained data from the execution of the test case to a performance profile for the application; and
providing the performance profile to a developer of the application.
17. The set of computer-executable instructions of clause 16, wherein the data characterizing the application to be evaluated further comprises one or more of a device type, an operating system for the device, a requested network slice type, and whether the application is expected to be bursty, continuous, bandwidth intensive, or latency sensitive with regards to network usage.
18. The set of computer-executable instructions of clause 16, wherein the instructions further cause the one or more electronic processors to generate a test case profile from the data characterizing the application, and the test case profile is generated using a trained model.
19. The set of computer-executable instructions of clause 16, wherein determining a network configuration for each test case further comprises determining one or more of bandwidth, network protocol, frequency band, and network slice for the test case.
20. The set of computer-executable instructions of clause 16, wherein the live network comprises user equipment, a radio, routing equipment, and one or both of a 5G and 4G core service, the deep packet inspection probe is a plurality of such probes, with each probe inserted at an interface in the live network, and the performance profile provided to the developer is a graph or table illustrating a consumption of a network resource by the application during the test case.
21. A system for evaluating the performance of an application when used with a network configuration, comprising:
a live telecommunications network;
a set of deep packet inspection probes installed at a plurality of interfaces of the live network;
a test generator operative to generate one or more test cases for the application based on data characterizing the application;
a network configuration element operative to configure the live network in a specific network configuration for testing the application;
a test case execution element operative to execute at least one of the generated test cases in the live network, where the network is configured in accordance with the specific network configuration;
a data collection element operative to collect data from the set of deep packet inspection probes;
a process to associate the collected data with performance of the application during execution of the test case; and
a process to generate one or more displays of the performance of the application during execution of the test case.
It should be understood that the present invention as described above can be implemented in the form of control logic using computer software in a modular or integrated manner. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and/or methods to implement the present invention using hardware and a combination of hardware and software.
In some embodiments, certain of the methods, models or functions described herein may be embodied in the form of a trained neural network or machine learning model, where the network or model is implemented by the execution of a set of computer-executable instructions. The instructions may be stored in (or on) a non-transitory computer-readable medium and executed by a programmed processor or processing element. The specific form of the method, model or function may be used to define one or more of the operations, functions, processes, or methods used in the development or operation of a neural network, the application of a machine learning technique or techniques, or the development or implementation of an appropriate decision process. Note that a neural network or deep learning model may be characterized in the form of a data structure in which are stored data representing a set of layers containing nodes, and connections between nodes in different layers are created (or formed) that operate on an input to provide a decision or value as an output.
In general terms, a neural network may be viewed as a system of interconnected artificial “neurons” or nodes that exchange messages between each other. The connections have numeric weights that are “tuned” during a training process, so that a properly trained network will respond correctly when presented with an image or pattern to recognize (for example). In this characterization, the network consists of multiple layers of feature-detecting “neurons”; each layer has neurons that respond to different combinations of inputs from the previous layers. Training of a network is performed using a “labeled” dataset of inputs in a wide assortment of representative input patterns that are associated with their intended output response. Training uses general-purpose methods to iteratively determine the weights for intermediate and final feature neurons. In terms of a computational model, each neuron calculates the dot product of inputs and weights, adds the bias, and applies a non-linear trigger or activation function (for example, using a sigmoid response function).
When implemented as a neural network, a machine learning model is a set of layers of connected neurons that operate to make a decision (such as a classification) regarding a sample of input data. A model is typically trained by inputting multiple examples of input data and an associated correct “response” or decision regarding each set of input data. Thus, each input data example is associated with a label or other indicator of the correct response that a properly trained model should generate. The examples and labels are input to the model for purposes of training the model. When trained (i.e., the weights connecting neurons have converged and become stable or within an acceptable amount of variation), the model will operate to respond to an input sample of data to generate a correct response or decision.
Any of the software components, processes or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as Python, Java, JavaScript, C++, or Perl using conventional or object-oriented techniques. The software code may be stored as a series of instructions, or commands in (or on) a non-transitory computer-readable medium, such as a random-access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM. In this context, a non-transitory computer-readable medium is almost any medium suitable for the storage of data or an instruction set aside from a transitory waveform. Any such computer readable medium may reside on or within a single computational apparatus and may be present on or within different computational apparatuses within a system or network.
According to one example implementation, the term processing element or processor, as used herein, may be a central processing unit (CPU), or conceptualized as a CPU (such as a virtual machine). In this example implementation, the CPU or a device in which the CPU is incorporated may be coupled, connected, and/or in communication with one or more peripheral devices, such as display. In another example implementation, the processing element or processor may be incorporated into a mobile computing device, such as a smartphone or tablet computer.
The non-transitory computer-readable storage medium referred to herein may include a number of physical drive units, such as a redundant array of independent disks (RAID), a floppy disk drive, a flash memory, a USB flash drive, an external hard disk drive, thumb drive, pen drive, key drive, a High-Density Digital Versatile Disc (HD-DV D) optical disc drive, an internal hard disk drive, a Blu-Ray optical disc drive, or a Holographic Digital Data Storage (HDDS) optical disc drive, synchronous dynamic random access memory (SDRAM), or similar devices or other forms of memories based on similar technologies. Such computer-readable storage media allow the processing element or processor to access computer-executable process steps, application programs and the like, stored on removable and non-removable memory media, to off-load data from a device or to upload data to a device. As mentioned, with regards to the embodiments described herein, a non-transitory computer-readable medium may include almost any structure, technology or method apart from a transitory waveform or similar medium.
Certain implementations of the disclosed technology are described herein with reference to block diagrams of systems, and/or to flowcharts or flow diagrams of functions, operations, processes, or methods. It will be understood that one or more blocks of the block diagrams, or one or more stages or steps of the flowcharts or flow diagrams, and combinations of blocks in the block diagrams and stages or steps of the flowcharts or flow diagrams, respectively, can be implemented by computer-executable program instructions. Note that in some embodiments, one or more of the blocks, or stages or steps may not necessarily need to be performed in the order presented or may not necessarily need to be performed at all.
These computer-executable program instructions may be loaded onto a general-purpose computer, a special purpose computer, a processor, or other programmable data processing apparatus to produce a specific example of a machine, such that the instructions that are executed by the computer, processor, or other programmable data processing apparatus create means for implementing one or more of the functions, operations, processes, or methods described herein. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement one or more of the functions, operations, processes, or methods described herein.
While certain implementations of the disclosed technology have been described in connection with what is presently considered to be the most practical and various implementations, it is to be understood that the disclosed technology is not to be limited to the disclosed implementations. Instead, the disclosed implementations are intended to cover various modifications and equivalent arrangements included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
This written description uses examples to disclose certain implementations of the disclosed technology, and to enable any person skilled in the art to practice certain implementations of the disclosed technology, including making and using any devices or systems and performing any incorporated methods. The patentable scope of certain implementations of the disclosed technology is defined in the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural and/or functional elements that do not differ from the literal language of the claims, or if they include structural and/or functional elements with insubstantial differences from the literal language of the claims.
All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and/or were set forth in its entirety herein.
The use of the terms “a” and “an” and “the” and similar referents in the specification and in the following claims are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “having,” “including,” “containing” and similar referents in the specification and in the following claims are to be construed as open-ended terms (e.g., meaning “including, but not limited to,”) unless otherwise noted. Recitation of ranges of values herein are merely indented to serve as a shorthand method of referring individually to each separate value inclusively falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments of the invention and does not pose a limitation to the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to each embodiment of the present invention.
As used herein (i.e., the claims, figures, and specification), the term “or” is used inclusively to refer to items in the alternative and in combination.
Different arrangements of the components depicted in the drawings or described above, as well as components and steps not shown or described are possible. Similarly, some features and sub-combinations are useful and may be employed without reference to other features and sub-combinations. Embodiments of the invention have been described for illustrative and not restrictive purposes, and alternative embodiments will become apparent to readers of this patent. Accordingly, the present invention is not limited to the embodiments described above or depicted in the drawings, and various embodiments and modifications can be made without departing from the scope of the claims below.
This application claims the benefit of U.S. Provisional Application No. 63/108,812, entitled “Systems and Methods for Optimization of Application Performance on a Telecommunications Network,” filed Nov. 2, 2020, the disclosure of which is incorporated, in its entirety (including the Appendices), by this reference.
| Number | Date | Country | |
|---|---|---|---|
| 63108812 | Nov 2020 | US |
