Cloud-based services for management of cell-based test systems

Abstract

Cloud-based management of cell-based test systems is described. For example, at a server that is communicatively coupled to a cell-based test system via a network (that is, in the cloud), a test case from a test case library is selected and accessed. The test case is selected according to the type of user equipment to be tested at the cell-based test system. The test case is sent from the server over the network to the cell-based test system. The cell-based test system can automatically perform the test case on the user equipment. Results from performing the test case on the user equipment are then stored on the cloud; that is, the test results are received at the server from the cell-based test system over the network and stored at the server. A report based on the test results can be prepared and stored on the server (in the cloud).

Description

RELATED U.S. APPLICATIONS

This application is related to the following U.S. patent applications, all of which are hereby incorporated by reference in their entirety: U.S. patent application Ser. No. 15/236,326, “Local Portable Test Systems and Methods,” by D. Doshi et al.; U.S. patent application Ser. No. 15/236,315, “Automated Configurable Portable Test Systems and Methods,” by D. Doshi et al.; U.S. patent application Ser. No. 15/236,314, “Automated Validation and Calibration Portable Test Systems and Methods,” by D. Doshi et al.; and U.S. patent application Ser. No. 15/236,277, “Device Profile-Driven Automation for Cell-Based Test Systems,” by D. Doshi et al.

BACKGROUND

In telecommunication applications, user equipment (UE) typically refers to devices such as smartphones, tablets, laptops, and other types of devices that provide end users with many different capabilities including the capability to communicate and exchange content with each other and with Web servers over a network. The definition of user equipment can be expanded to also include the Internet-of-things (IoT), which are devices that provide the capability to collect and exchange data amongst themselves and with servers over a network. User equipment can communicate over networks through a wired or wireless medium using different types of network protocols such as but not limited to the 3rd Generation Partnership Project (3GPP) Long-Term Evolution (LTE) standard.

The number of UE devices is growing rapidly. New types of devices and new versions of existing types of devices, with diverse capabilities, are continuously and rapidly being introduced. Thorough testing of the devices under many different scenarios is essential to verify that the devices will function correctly.

Automated testing is usually relied upon because it is scalable to large numbers of devices and can be accomplished quicker, more systematically, and at less expense than, for example, manual testing. This is turn can increase test coverage and reduce time-to-market, increasing reliability while reducing the cost to both manufacturers and consumers.

However, there are aspects of conventional automated testing paradigms that are costly. Automated testing is currently performed in large-scale, centralized laboratories that include a shielded room that prevents radio frequency (RF) signals that might interfere with the testing from entering or escaping. Devices to be tested are shipped in quantity to the laboratories, which can contribute to the cost of testing. While automated to a large degree, aspects of the testing still require significant manual support, such as equipment maintenance, test case delivery, device profile delivery, test data collection, data analytics and reporting, and consulting, for example. These factors contribute significantly to the cost of testing.

SUMMARY

Instead of large-scale, centralized laboratories, embodiments according to the present invention introduce smaller, cell-based test systems that can be deployed at many different locations such as original equipment manufacturer (OEM) sites and operator laboratories. Each test system includes one or more portable test units (e.g., shielded boxes) that can be communicatively coupled to a local control component (e.g., a computer system). To test some types of devices such as smartphones, the cell-based test systems can each also include one or more local test user equipment interface components (e.g., base stations). The user equipment to be tested can be placed inside a portable test unit, and automated testing is performed under control of software that executes on the computer system.

The computer system that executes the testing, or another computer system communicatively coupled to the first computer system, can upload and download information to a centralized server via a network (e.g., the Internet). In other words, embodiments according to the invention utilize cloud-based services to manage the cell-based test systems individually and collectively.

Relative to conventional test laboratories, the portability of the cell-based test systems means that they are less costly, more scalable, more quickly deployed and updated, and allow more flexible test coverage. The use of cloud-based services to manage the cell-based test systems increases the degree of test automation and decreases the amount of manual support needed. Testing can be performed at manufacturing sites, so that devices to be tested do not have to be shipped to testing laboratories. Overall, the cell-based test systems can reduce time-to-market and overall cost without affecting test coverage and product reliability.

In an embodiment, at a server that is communicatively coupled to a cell-based test system via a network (that is, at a server in the cloud), a test case from a test case library is selected and accessed. The test case is selected according to profile information about user equipment to be tested at the cell-based test system (e.g., the type, make, model, and/or features of the user equipment).

The test case is sent from the server over the network to the cell-based test system. The cell-based test system can automatically perform the test case on the user equipment. Results from performing the test case on the user equipment can then be stored in the cloud; that is, the test results can be received at the server from the cell-based test system over the network and stored at the server. A report based on the test results can be prepared and stored on the server (in the cloud), and can be accessed and downloaded. Such a process can be performed for multiple, different test cases at each of the cell-based test systems.

In an embodiment, results from performing test cases on different types, makes, models, and features of user equipment at multiple cell-based test systems are received at the server from the various test systems over the network. In such an embodiment, the results for the different types and models of user equipment can be compared and summarized. In other words, test results from different cell-based test systems can be uploaded to the cloud and compared, and the results of the comparison can be downloaded from the cloud.

Embodiments according to the invention can perform other cloud-based services to manage the cell-based test systems. In an embodiment, at the server (in the cloud), a device profile from a device profile library is selected and accessed. The selected device profile corresponds to the profile information about the user equipment to be tested at the cell-based test system. The device profile is sent from the server to the cell-based test system over the network (the test system downloads the device profile from the cloud).

In an embodiment, reference user equipment is used to calibrate the cell-based test system. In such an embodiment, information about characteristic(s) of the cell-based test system measured using the reference user equipment is received and stored at the server from the cell-based test system over the network (the information about the characteristic(s) is uploaded to and stored in the cloud by the cell-based test system). Prior to the cell-based test system performing the test case on the user equipment, that information is sent from the server over the network to the cell-based test system (the information is downloaded from the cloud by the cell-based test system) and can be used to calibrate the cell-based test system.

In an embodiment, software and software updates that can be used by the computer system to perform automated testing are sent from the server over the network to the cell-based test system (the software is downloaded from the cloud by the cell-based test system).

In an embodiment, licensing information for the cell-based test system is stored at the server (in the cloud) and can be accessed and downloaded.

In an embodiment, access information that is used to control access to, for example, the test results stored at the server is also stored at the server.

Cloud-based services in embodiments according to the invention provide a number of benefits and advantages. Test results can be quickly and securely stored and accessed. Test cases can be quickly delivered on demand. The correct device profiles can be quickly identified and delivered. Software updates and new device profiles can be developed and quickly distributed. License agreements can be prepared and quickly implemented. Testing can be made more robust and feature-rich.

These and other objects and advantages of the various embodiments according to the present invention will be recognized by those of ordinary skill in the art after reading the following detailed description of the embodiments that are illustrated in the various drawing figures.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and form a part of this specification and in which like numerals depict like elements, illustrate embodiments of the present disclosure and, together with the detailed description, serve to explain the principles of the disclosure.

FIG. 1 is a block diagram illustrating an example of a network or system that can be used to implement cloud-based services for managing cell-based test systems in an embodiment according to the invention.

FIG. 2 illustrates elements of a cell-based test system in an embodiment according to the invention.

FIG. 3A is a block diagram illustrating elements of a server that provides cloud-based services for managing cell-based test systems in an embodiment according to the invention.

FIG. 3B illustrates examples of the types of information that are exchanged between authorized users, a server in the network (cloud), and a cell-based test system in an embodiment according to the invention.

FIG. 4 is a diagram illustrating an example of certification testing performed using a cell-based test system.

FIG. 5 is a diagram illustrating another example of a test using a cell-based test system.

FIG. 6 is a flowchart of a computer-controlled method of managing cell-based test systems in an embodiment according to the present invention.

FIG. 7 is a flowchart of a computer-controlled method of testing user equipment at a cell-based test system in an embodiment according to the present invention.

FIG. 8 is a block diagram illustrating elements of a computer system in an embodiment according to the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to the various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. While described in conjunction with these embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

Some portions of the detailed descriptions that follow are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those utilizing physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as transactions, bits, values, elements, symbols, characters, samples, pixels, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present disclosure, discussions utilizing terms such as “receiving,” “storing,” “sending,” “selecting,” “accessing,” “comparing,” “using,” “determining,” “generating,” “performing,” “controlling,” “executing,” “using,” “installing,” “initializing,” “configuring,” or the like, refer to actions and processes (e.g., flowcharts 600 and 700 of FIGS. 6 and 7, respectively) of an apparatus or computer system or similar electronic computing device or processor (e.g., the computer system 800 of FIG. 8). A computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within memories, registers or other such information storage, transmission or display devices.

Embodiments described herein may be discussed in the general context of computer-executable instructions residing on some form of computer-readable storage medium, such as program modules, executed by one or more computers or other devices. By way of example, and not limitation, computer-readable storage media may comprise non-transitory computer storage media and communication media. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.

Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory (e.g., an SSD) or other memory technology, compact disk ROM (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can accessed to retrieve that information.

Communication media can embody computer-executable instructions, data structures, and program modules, and includes any information delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media. Combinations of any of the above can also be included within the scope of computer-readable media.

FIG. 1 is a block diagram illustrating an example of a network or system 100 that can be used to implement cloud-based services for managing cell-based test systems in an embodiment according to the invention. In the example of FIG. 1, the system 100 includes a remote management environment that includes a number of servers 102a-102n, and also includes a local test environment that includes a number of cell-based test systems 104a-104m. For simplicity, the following discussion may refer to a single server 102a and a single cell-based test system 104a, but that discussion can be readily extrapolated to the use of multiple servers and multiple cell-based test systems. The servers 102a-102n and the cell-based test systems 104a-104m are communicatively coupled (wired or wirelessly) via a network 110 (e.g., the public Internet, but not so limited). The network 110 may be a cloud-enabled network or a cloud-based network.

FIG. 2 illustrates elements of a cell-based test system 104a in an embodiment according to the invention. The cell-based test system 104a includes one or more local test control components, one or more local test user equipment interface components, and one or more portable test units.

In an embodiment, each local test control component is implemented on a computer system 210. In an embodiment, the local test control component is implemented as a virtual machine on the computer system 210. The computer system 210 can communicate with (is communicatively coupled to) the network (cloud) 110 to enable and facilitate the use of cloud-based services to manage the cell-based test system 104a, as will be described more fully below.

In general, a local test control component is configured to direct the local test user equipment interface component(s) and to control test interactions with the user equipment 230a-230k to be or being tested, and a local test user equipment interface component is operable for communicating with the user equipment during testing. The local test control component on the computer system 210 is used for test automation, including test case execution automation, user equipment automation, and test equipment automation. The local control component on the computer system 210 is also used for collecting and reporting test results to the server 102a, and may analyze and present (e.g., display) the test results. In an embodiment, the computer system 210 is or emulates an Internet Protocol (IP) Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) server as described by 3GPP. These and other functions of the computer system 210 are presented further below.

In an embodiment, a second computer system 212 (e.g., a laptop) is used to implement and execute a network control plane simulator or test executive. In an embodiment, the test executive is an Evolved Packet Core (EPC), also known as a System Architecture Evolution (SAE) Core. The EPC can be used for testing voice-and-data devices (e.g., smartphones) and operates in accordance with contemporary standard(s) appropriate for the type of user equipment being tested, such as the 3rd Generation Partnership Project (3GPP) Long-Term Evolution (LTE) standard. A test executive or network control plane simulator for other, non-3GPP systems such as WiFi™, WIMAX (Worldwide Interoperability for Microwave Access), and CDMA2000 can also be implemented on the computer system 212.

In an embodiment, the computer system 210 and the computer system 212 communicate over an Ethernet connection. In another embodiment, the test executive (e.g., the EPC) is implemented on the computer system 210 instead of the second computer system 212. For example, the test executive (e.g., the EPC) can be executed as a virtual machine on the computer system 210.

In an embodiment, each local test user equipment interface component is a network access point simulation component implemented as, for example, a base station 220. In one such embodiment, a base station 220 is or simulates an LTE Evolved Node B (eNodeB). In general, a base station 220 is implemented as a small cell or femtocell. A base station 220 can have both wired and wireless connections to connect to the user equipment 230a-230k.

As noted above, each cell-based test system can include one or more local test user equipment interface components (one or more base stations). Test configurations in which multiple base stations are connected to one test executive (e.g., EPC) and in which one base station is connected to one test executive (e.g., EPC) are supported.

In an embodiment, the base station 220 communicates with the test executive (e.g., the EPC) via an Ethernet connection. The base station 220 and the test executive (e.g., the EPC) can be operated in a normal mode (e.g., to support a normal call flow) or error conditions may be injected, depending on the requirements of the test cases.

One or more items of user equipment (UEs) 230a-230k can be communicatively coupled (wired or wirelessly) to each base station in the cell-based test system 104a. In an embodiment, there are 32 UEs coupled to each base station, although the invention is not so limited. The UEs (which also may be referred to as devices under test) can be, but are not limited to, telecommunication devices (e.g., smartphones) and Internet-of-things (IoT) devices (e.g., sensors and meters).

In an embodiment, the user equipment to be tested is placed inside a portable test unit. In an embodiment, the portable test unit is implemented as a shielded box 240. Like the other elements of the cell-based test system 104a, the shielded box 240 is portable. For example, the shielded box 240 can be about 24 by 24 by 24 inches in size.

The shielded box 240 illustrated in FIG. 2 has an open front to show the UEs 230a-230k that are inside. However, during testing, the opening is closed by a cover or door (not shown). The shielded box 240 allows radio frequency (RF) signals to pass back-and-forth between the base station 220 and the user equipment 230a-230k inside the shielded box. The shielded box 240 can also include pass-through connectors (e.g., Universal Serial Bus connectors) that allow the base station 220 to communicate with the user equipment 230a-230k in the shielded box using cables. Similar connectors on the shielded box 240 can also be used to connect power cables to the user equipment 230a-230k inside the shielded box.

FIG. 3A is a block diagram illustrating elements of a server 102a that provides cloud-based services for managing cell-based test systems in an embodiment according to the invention. The server 102a can be communicatively coupled to the cell-based test systems 104a-104m over the public Internet (the network 110, or cloud).

In the example of FIG. 3A, the server 102a includes the following modules: a test case library 302, a device profile library 304, a test equipment manager 306, a test user manager 308, a test data repository 310, a report and analytics module 312, and a subject matter expert (SME) library 314. The functions associated with each of these modules are presented below. While these elements and their functionality are described as separate modules, the modules can be combined in any manner.

The test case library 302 includes a library of different test cases that can be executed using the cell-based test systems 104a-104m. Test cases may also be known as user equipment conformance tests. A test case may specify the purpose of the test, the standards-based requirement being tested, a procedure for performing the test, starting conditions, and pass criteria. As new test cases are developed, they can be readily added to the cloud-based test case library 302. Standard industry test cases are specified by organizations such as 3GPP. Customized test cases may be specified by users (e.g., manufacturers or operators) or developed for users based on their test requirements and specifically for their products.

The cell-based test system 104a, for example, can send profile information to the server 102a that identifies, for example, the type, make, model, and/or features of user equipment to be tested. Based on the profile information, the server 102a selects one or more test cases appropriate to the user equipment to be tested. The server 102a can then access the selected tests cases in the test case library 302 and send them over the network 110 to the cell-based test system 104a. That is, the selected test case or test cases are downloaded from the cloud to the cell-based test system 104a. The cell-based test system 104a can then automatically perform the test case(s) on the user equipment. Test results can be stored by the cell-based test system 104a (at least temporarily), then securely uploaded via the cloud (over the network 110) to the server 102a and stored in the test data repository 310.

The device profile library 304 includes a library of different device profiles that can be executed on the cell-based test systems 104a-104m. Device profiles are developed and validated and then stored in the cloud (in the device profile library 304 on the server 102a), and then downloaded for testing.

More specifically, the cell-based test system 104a can send profile information to the server 102a that identifies, for example, the type, make, model, and/or features of the user equipment to be tested. In response, the device profile corresponding to the profile information (that is, appropriate for the, e.g., type, make, model, and/or features of the user equipment to be tested) is selected and accessed in the device profile library 304 (FIG. 3A) and sent from the server 102a over the network 110 to the cell-based test system 104a. That is, the selected device profile is downloaded from the cloud to the cell-based test system 104a. The cell-based test system 104a can then automatically use the device profile to perform the test case(s) on the user equipment.

A device profile has the intelligence that is required to control a given type of user equipment being tested. A device profile may be a standard application program interface (API) or an API adapted by an original equipment manufacturer (OEM). In essence, a device profile is or includes a test script that adapts the generalized software (test code) executed by the computer system 210 to the specific type, make, model, and/or features of the user equipment being tested.

More specifically, at the cell-based test system 104a, the device profile (e.g., API) is downloaded to the computer system 210, receives a command from the server, adapts the command to the user equipment 230a-230k being tested, executes the command on the user equipment, and responds to the computer system 210 with the result of the command execution. The device profile has the intelligence to automatically switch from a first automation mode that uses commands for a first operating system (e.g., Android™) to a second automation mode that uses commands for a second, different operating system (e.g., iOS). For example, one automation mode may use the ADB command, while another automation mode may use the AT command, in which case the device profile has the intelligence to determine which command to use and then to properly adapt the command from the computer system 210 according to the type of command used by the automation mode.

As new types, makes, models and/or features of devices are introduced, new device profiles can be written, validated, and added to the device profile library 304. Users can also modify a device profile and upload the modification and/or the modified device profile to the server 102a. Once validated, the modified device profile can be added to the device profile library 304 and shared with other users to improve the test process for the other users. In this manner, the cloud-based device profile library 304 acts like a marketplace for new and improved automation details. Alternatively, the modified device profile can be kept private to the user that developed it.

The test equipment manager 306 stores, issues, and transmits test software (e.g., the test executive (e.g., the EPC)) to the cell-based test systems 104a-104m (specifically, to the computer system 210). That is, the software is downloaded from the cloud to the cell-based test systems 104a-104m. The test equipment manager 306 can also store, issue, and transmit (download) test configuration information, which can include user equipment control configuration information, to the cell-based test systems 104a-104m.

The test equipment manager 306 can also deliver (download) updates to the software (e.g., the test executive (e.g., the EPC)) already present at the cell-based test systems 104a-104m. Updates can be downloaded to each of the cell-based test systems 104a-104m. In an embodiment, information about which updates to download and apply is embedded in the software applications, so that the applications and/or an authorized user can determine which updates to apply and when to apply them. In another embodiment, the server 102a accesses the cell-based test systems 104a-104m to determine, for example, what software and software version is present on each test system, then determines what additional software, or what software updates, need to sent to each test system, and then automatically downloads the appropriate software and/or software updates to the respective test systems. In an embodiment, the server 102a (e.g., the test equipment manager 306) maintains a record or log of the software present on each of the cell-based test systems 104a-104m and uses that information to determine whether and what type of software and/or software update is needed. Also, software updates (such as a patch) that are appropriate across all of the cell-based test systems 104a-104m can be automatically downloaded to the test systems.

In an embodiment, configuration of the local test control component and local test user equipment interface on the cell-based test systems 104a is automated using the information downloaded from the test equipment manager 306. Software and software updates—including the test executive (e.g., the EPC), test cases, device profiles, etc.—are automatically installed and initialized. Software and software updates for configuring the computer system 210 and/or the computer system 212 according to the given configuration can also be automatically installed, configured, and initialized. Calibration can then be automatically performed, and test cases can be executed to verify successful installation and operation.

In an embodiment, reference user equipment (a reference component) is used to calibrate the cell-based test system 104a. The reference user equipment is a trusted component with reliable and known communication characteristics and features. The reference user equipment can be configured to simulate at least a portion of the functionality of the user equipment 230a-230k to be tested. In such an embodiment, information about characteristic(s) of the cell-based test system 104a measured using the reference user equipment is received and stored by the test equipment manager 306. That is, the information about the characteristic(s) of the cell-based test system 104a is uploaded to and stored in the cloud. Prior to the cell-based test system 104a performing a test case or test cases on the user equipment 230a-230k to be tested, that information is sent from the server 102a over the network 110 to the cell-based test system (the information is downloaded from the cloud to the cell-based test system) and used before and during testing for calibration purposes, for example.

For example, signal loss between the base station 220 and the reference user equipment can be measured, then uploaded to and stored in the cloud. Prior to testing, the signal loss information can be downloaded to the cell-based test system 104a. During testing, the cell-based test system 104a can be calibrated to properly compensate for the signal loss by appropriately varying the signal strength between the base station 220 and the user equipment 230a-230k being tested. Other characteristics besides signal loss can be measured and uploaded to the cloud (e.g., the server 102a). Those characteristics can be generally referred to as physical layer characteristics, protocol characteristics, and data characteristics. In addition to signal loss, physical layer characteristics include, but are not limited to, RF signal power, RF signal frequency, and bandwidth. Protocol characteristics include, but are not limited to, scheduling and security aspects. Data characteristics include, but are not limited to, data throughput. In general, these characteristics can be measured, the measurement results can be uploaded to and stored in the cloud, and the results can be subsequently downloaded to calibrate the cell-based test system 104a prior to and during testing.

The test user manager 308 can manage account information for each of the users of the cell-based test system 104a. A “user” may be, for example, an entity that has purchased, licensed, or leased a cell-based test system or an authorized representative of such an entity. The test user manager 308 can store and enforce access information (e.g., user identifiers and passwords) that is used to control access to the account information. The account information may include licensing information, test data/results and reports (stored in the test data repository 310), and other information. The account information, in addition to being accessed on the on the server 102a (via the cloud), can be downloaded to the cell-based test systems 104a-104m and/or to a computer system or server designated by the user. For example, if a user purchases a time-based extension to their current license, such a purchase can be made online via the test user manager 308, and the user can then download a new license file from the test user manager.

Results from performing the test case(s) on the user equipment are stored in the cloud in the test data repository 310 on the server 102a. That is, test results are received at the server 102a from the cell-based test system 104a over the network 110 and stored at the server. The test results stored in the test data repository 310 can be accessed by authorized users as noted, and can also be downloaded to any computer system or server designated by the user. A report based on the test results can be prepared and stored in the test data repository 310 on the server 102a (in the cloud) and subsequently accessed and downloaded by authorized users.

In an embodiment, results from performing test cases on different types, makes, models, and features of user equipment at multiple cell-based test systems are received at the server 102a from those test systems over the network 110. In such an embodiment, the report and analytics module 312 compares and summarizes the results for the different user equipment. Thus, test results from different cell-based test systems can be uploaded to the cloud and compared, and the results of the comparisons can be subsequently accessed and downloaded by authorized users.

In an embodiment, the SME library 314 is available to be readily accessed for consultation by authorized users via the cloud (over the network 110) and downloaded. Authorized users can also add new information to the SME library 314 and update existing information.

FIG. 3B illustrates examples of the types of information that are exchanged between authorized users, a server 102a in the network (cloud) 110 (FIG. 1), and a cell-based test system 104a in an embodiment according to the invention.

For example, the cell-based test system 104a sends profile information for the UEs 230a-230k to be tested to the server 102a. In response, based on the profile information, the server 102a sends test case(s) to the cell-based test system 104a. In an embodiment, the server 102a also sends device profile(s) to the cell-based test system 104a. Also, an authorized user can add device profile(s) to the server 102a and/or update existing device profile(s).

The server 102a can send test software and software updates to the cell-based test system 104a.

After operation of the cell-based test system 104a with a reference UE (after characteristics of the test system are measured using the reference UE), the test system can send information about the characteristics (the results of the measurements) to the server 102a. The server 102a can subsequently send the information about the characteristics of the cell-based test system 104a to the test system when testing is to be performed.

The server 102a can establish accounts with authorized users and issue access information (e.g., passwords) to users. Users use the access information to log into the server 102a, and can access and download account information including, but not limited to, licenses and licensing information, test results, analytics, and reports, and can also access the SME library. Also, an authorized user can add device profile(s) to the server 102a and/or update existing device profile(s).

FIG. 4 is a diagram illustrating an example of certification testing performed using the cell-based test system 104a. In this example, the UE 230a is a smartphone.

At operation 1, the computer system 210 sends a “turn airplane mode on” command to the UE 230a. At operation 2, the computer system 210 sends a “turn airplane mode off” command to the UE 230a.

At operation 3, the UE 230a accomplishes LTE detach followed by LTE attach. At operation 4, the computer system 210 instructs the base station 220 to verify successful LTE attach.

At operation 5, the UE 230a registers with an IMS server 402 (which may be simulated by the computer system 210). At operation 6, the computer system 210 instructs the base station to verify successful IMS registration.

At operation 7 of FIG. 4, the computer system 210 sends an AT command (plus COPN) to the UE 230a to get operator names. At operation 8, the computer system 210 sends, for example, an AT command (plus CGPIAF) to the UE 230a to print the IP address format. At operation 9, the computer system 210 sends, for example, an AT command (plus CGPIAF) to the UE 230a to modify the IP address format. Other types of commands, such as ADB commands, can be used instead of AT commands.

FIG. 5 is a diagram illustrating an example of a test for application retry using the cell-based test system 104a. In this example, the UE 230a is an IoT device.

At operation 1, the computer system 210 instructs the base station 220 to configure an LTE band. At operation 2, the computer system 210 instructs the base station 220 to configure an Internet path.

At operation 3, the UE 230a is powered on by the computer system 210.

At operation 4, if applicable, the UE 230a accomplishes LTE attach and registration with an IMS server (not shown).

At operation 5, a path to the IoT server 502 (which may be simulated by the computer system 210) is established by the UE 230a. At operation 6, the computer system 210 instructs the base station 220 to monitor IP connectivity and pattern.

At operation 7 of FIG. 5, the UE 230a auto-registers with the IoT server 502. At operation 8, the computer system 210 instructs the base station 220 to monitor registration activity and pattern.

FIG. 6 is a flowchart 600 of a method of managing the cell-based test systems 104a-104m of FIG. 1 in an embodiment according to the present invention. All or some of the operations represented by the blocks in the flowchart 600 can be implemented as computer-executable instructions residing on some form of non-transitory computer-readable storage medium, and performed by a computer system such as the server 102a of FIG. 1.

FIG. 6 is described with reference also to FIG. 3A. Where certain elements are described in the singular in the following discussion, such discussion can be readily extended to the plural. While the blocks in the flowchart 600 are presented in a certain order, the operations described by those blocks may be performed in a different order.

In block 602, at the server 102a, a test case from a test case library 302 is selected and accessed. The test case is selected according to profile information (e.g., type, make, model, and/or features) about the user equipment 230a-230k to be tested at a cell-based test system 104a. The profile information can be sent from the cell-based test system 104a to the server 102a over the network 110.

In block 604, the selected test case is sent from the server 102a over the network 110 to the cell-based test system 104a.

In block 606, at the server 102a, a device profile from a device profile library 304 is selected and accessed. The selected device profile corresponds to the profile information about the user equipment 230a-230k to be tested at the cell-based test system 104a.

In block 608, the device profile is sent from the server 102a to the cell-based test system 104a over the network 110.

In an embodiment, reference user equipment is used to calibrate the cell-based test system 104a. In such an embodiment, in block 610, information about characteristics (e.g., the physical, protocol, and data characteristics mentioned above) of the cell-based test system 104a is received from the cell-based test system over the network 110 and stored by the test equipment manager 306 at the server 102a. Prior to the cell-based test system 104a performing the test case on the user equipment 230a-230k, that information is sent from the server 102a over the network 110 to the cell-based test system, in block 612. That information can be used to calibrate the cell-based test system 104a prior to and during testing of the user equipment 230a-230k.

In block 614, in an embodiment, software and/or software updates that can be used by the cell-based test system 104a to perform automated testing are sent from the server 102a (specifically, the test equipment manager 306) over the network 110 to the cell-based test system.

The cell-based test system 104a can then automatically perform the test case on the user equipment 230a-230k.

In block 616, results from performing the test case on the user equipment 230a-230k are received at the server 102a from the cell-based test system 104a over the network 110 and stored at the server in the test data repository 310. A report based on the test results can be prepared and stored on the server 102a in the test data repository 310, and can be accessed by authorized users. In an embodiment, access information that is used to control access to, for example, the test results stored at the server 102a is also stored at the server by the test user manager 308.

In an embodiment, in block 618, results from performing test cases on different types, makes, models, and features of user equipment at multiple cell-based test systems are received at the server 102a from those systems over the network 110. In such an embodiment, the results for the different types and models of user equipment can be compared and summarized in a report that is prepared and stored on the server 102 by the report and analytics module 312. The report is accessible to authorized users under control of the test user manager 308.

FIG. 7 is a flowchart 700 of a method of testing user equipment (e.g., the user equipment 230a-230k of FIG. 2) at a cell-based test system (e.g., the cell-based test system 104a of FIG. 2) in an embodiment according to the present invention. All or some of the operations represented by the blocks in the flowchart 700 can be implemented as computer-executable instructions residing on some form of non-transitory computer-readable storage medium, and performed by a computer system such as the computer system 210 and test executive (e.g., the EPC) of FIG. 2.

FIG. 7 is described with reference also to FIG. 3A. Where certain elements are described in the singular in the following discussion, such discussion can be readily extended to the plural. While the blocks in the flowchart 700 are presented in a certain order, the operations described by those blocks may be performed in a different order.

In block 702, the user equipment 230a-230k to be tested is connected to the cell-based test system 104a. The cell-based test system 104a is then configured and initialized. In an embodiment, before the user equipment 230a-230k is connected to the cell-based test system 104a, characteristics of the cell-based test system are measured and sent to the server 102a over the network 110.

In block 704, profile information (e.g., type, make, model, and/or features) for the user equipment 230a-230k is detected/determined.

In block 706, the profile information is sent to the server 102a over the network 110.

In block 708, a device profile that is selected according to the profile information is received from the server 102a over the network 110. A user (e.g., test operator) can confirm usage of the selected device profile for test automation.

In block 710, a test case is received from the server 102a over the network 110. The test case is selected based on the profile information sent to the server 102a.

In block 712, in an embodiment, the information about the characteristics of the cell-based test system 104a is received from the server 102a over the network 110.

In block 714, the test case is performed on the user equipment 230a-230k. During performance of the test case, in an embodiment, the information about the characteristics of the cell-based test system 104a that is received from the server 102a can be used to calibrate the cell-based test system 104a. During performance of the test case, the user equipment 230a-230k is controlled according to the device profile in response to the software executing on the computer system 210. Thus, the information from the downloaded test profile is used by the cell-based test system 104a to perform appropriate test automation during execution of the test case.

In block 716, results from performing the test case on the user equipment 230a-230k are sent to the server 102a over the network 110.

In block 718, in an embodiment, a modification to the device profile is received from a user. The modification can be sent to the server 102a over the network 110.

Embodiments according to the invention utilize cloud-based services to manage cell-based test systems individually and collectively. The use of cloud-based services increases the degree of test automation and decreases the amount of manual support needed. Testing can be performed at manufacturing sites, so that devices to be tested do not have to be shipped to testing laboratories.

Test results can be quickly and securely stored and accessed. Test cases can be quickly delivered on demand. The correct device profiles can be quickly identified and delivered. Software updates and new device profiles can be developed and quickly distributed. License agreements can be prepared and quickly implemented. Testing can be made more robust and feature-rich.

Overall, time-to-market and overall cost are reduced without affecting test coverage and product reliability.

FIG. 8 is a block diagram illustrating elements of a computer system 800 in an embodiment according to the present invention. In an embodiment, the computer system 800 represents a platform for implementing the server 102a of FIG. 1, and the computer system 210 and computer system 212 of FIG. 2.

The computer system 800 broadly includes any single or multi-processor computing device or system capable of executing computer-readable instructions. In its most basic configuration, the computer system 800 may include at least one processing circuit (processor) 802 and at least one storage medium 804. The processor 802 generally represents any type or form of processing unit or circuit capable of processing data or interpreting and executing instructions. In an embodiment, the processor 802 receives instructions from a software application or module. The storage medium 804 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or other computer-readable instructions.

The computer system 800 may also include one or more components or elements in addition to the processor 802 and storage medium 804. For example, the computer system 800 may include an input/output (I/O) device 806, such as a keyboard and mouse, and a communication interface 808, each of which may be interconnected via a communication infrastructure (e.g., a bus 810). The computer system 800 may also include a display device 812.

The communication interface 808 may be any type or form of communication device or adapter capable of facilitating communication between the computer system 800 and one or more other devices including a network of other devices. The communication interface 808 can include, for example, a receiver and a transmitter that can be used to receive and transmit information (wired or wirelessly).

While the foregoing disclosure sets forth various embodiments using specific block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and/or component described and/or illustrated herein may be implemented, individually and/or collectively, using a wide range of hardware, software, or firmware (or any combination thereof) configurations. In addition, any disclosure of components contained within other components should be considered as examples because many other architectures can be implemented to achieve the same functionality.

The process parameters and sequence of steps described and/or illustrated herein are given by way of example only and can be varied as desired. As mentioned above, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. Also, the various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

While various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. These software modules may configure a computing system to perform one or more of the example embodiments disclosed herein. One or more of the software modules disclosed herein may be implemented in a cloud computing environment. Cloud computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a Web browser or other remote interface. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the disclosure is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the disclosure.

Embodiments according to the invention are thus described. While the present disclosure has been described in particular embodiments, it should be appreciated that the invention should not be construed as limited by such embodiments, but rather construed according to the following claims.

Claims

1. A method of managing a plurality of cell-based test systems for user equipment, the method comprising: selecting and accessing, at a server communicatively coupled to the plurality of cell-based test systems via a network, a test case from a test case library, wherein the user equipment comprises user equipment to be tested at a cell-based test system of the plurality of cell-based test systems and also comprises reference user equipment that is operable for calibrating the cell-based test system, wherein the test case is selected according to profile information about the user equipment to be tested, wherein the cell-based test system comprises a portable test unit and a computer system coupled to the portable test unit, and wherein the portable test unit comprises a shielded box configured to contain the user equipment to be tested;sending the test case, from the server over the network, to the cell-based test system, wherein the cell-based test system is operable for automatically performing the test case on the user equipment to be tested;selecting and accessing, at the server, a device profile from a device profile library, wherein the device profile corresponds to the profile information about the user equipment to be tested and comprises an application program interface operable for controlling the user equipment to be tested;sending the device profile from the server over the network to the cell-based test system;receiving, at the server over the network from the cell-based test system, information about a characteristic of the cell-based test system measured using the reference user equipment;storing the information about the characteristic at the server;prior to the cell-based test system performing the test case on the user equipment to be tested, sending the information about the characteristic from the server over the network to the cell-based test system;receiving, at the server from the cell-based test system over the network, results from performing the test case on the user equipment coupled to the cell-based test system; andstoring the results at the server.

2. The method of claim 1, further comprising sending software executable by the computer system from the server over the network to the cell-based test system.

3. The method of claim 1, further comprising storing, at the server, licensing information for the cell-based test system, wherein the licensing information is accessible and downloadable by an authorized user.

4. The method of claim 1, further comprising storing, at the server, access information configured to control access to the results stored at the server.

5. The method of claim 1, further comprising generating a report based on the results from performing the test case, wherein the report is accessible and downloadable by an authorized user.

6. The method of claim 1, further comprising: receiving, at the server from the plurality of cell-based test systems over the network, results from performing test cases on different types, makes, and models of user equipment coupled to the cell-based test systems, wherein the results for the different types, makes, and models of user equipment are accessible and downloadable by an authorized user; andperforming a comparison of the results for the different types, makes, and models of user equipment and providing a summary of the comparison, wherein the summary is accessible and downloadable by an authorized user.

7. A system comprising a server in a network wherein the server interacts with cell-based test systems, the system comprising: a processor;a communication interface coupled to the processor and operable for communicating with a plurality of the cell-based test systems over the network; anda memory coupled to the processor, wherein the memory has stored therein a plurality of modules that, when executed, perform operations for managing the plurality of the cell-based test systems, the modules comprising: a test case library comprising a plurality of test cases for different types of user equipment, wherein the user equipment comprises user equipment to be tested at a cell-based test system of the plurality of cell-based test systems and also comprises reference user equipment that is operable for calibrating the cell-based test system, wherein a test case is selected and accessed from the test case library according to profile information for user equipment to be tested at a cell-based test system of the plurality of the cell-based test systems and sent to the cell-based test system over the network, wherein the cell-based test system comprises a portable test unit and a computer system coupled to the portable test unit, wherein the portable test unit comprises a shielded box configured to contain the user equipment to be tested, and wherein the cell-based test system is operable for automatically performing the test case on the user equipment to be tested;a device profile library, wherein a device profile is selected and accessed from the device profile library according to the profile information about the user equipment to be tested and wherein the device profile is sent to the cell-based test system over the network, and wherein the device profile comprises a test script operable for controlling the user equipment to be tested;a test equipment manager operable for receiving information about a characteristic of the cell-based test system measured using the reference user equipment, for storing the information about the characteristic, and for sending the information about the characteristic over the network to the cell-based test system prior to the cell-based test system performing the test case on the user equipment to be tested; anda test data repository operable for storing results from performing the test case, wherein the results are received at the server from the cell-based test system over the network, wherein the results are accessible and downloadable by an authorized user.

8. The system of claim 7, wherein the test equipment manager further comprises software executable by the computer system, wherein further the server is operable for sending the software over the network to the cell-based test system and for sending updates to the software over the network to the cell-based test system.

9. The system of claim 7, wherein the plurality of modules further comprise a test user manager comprising licensing information for the cell-based test system, wherein the licensing information is accessible and downloadable by an authorized user.

10. The system of claim 9, wherein the test user manager further comprises test user account information for controlling access to information on the server.

11. The system of claim 7, wherein the plurality of modules further comprise a report and analytics module operable for generating a report based on the results from performing the test case, wherein the report is accessible and downloadable by an authorized user.

12. The system of claim 11, wherein the results in the test data repository comprise results from performing test cases on different types, makes, and models of user equipment coupled to the plurality of cell-based test systems, wherein the report and analytics module is further operable for performing a comparison of the results for the different types, makes, and models of user equipment and providing a summary of the comparison, wherein the summary is accessible and downloadable by an authorized user.

13. A system comprising a cell-based test system for user equipment and that interacts with a server, the system comprising: a portable test unit comprising a shielded box; anda computer system coupled to the portable test unit and operable for communicating with the server over a network, wherein the computer system is further operable for performing operations comprising: sending, to the server over the network, profile information describing the user equipment, wherein the user equipment comprises user equipment to be tested at the cell-based test system and also comprises reference user equipment that is operable for calibrating the cell-based test system;receiving, from the server over the network, a device profile that corresponds to the profile information about the user equipment to be tested and comprises an application program interface operable for controlling the user equipment to be tested;receiving a test case from the server over the network, wherein the test case is selected according to the profile information;sending, to the server over the network, information about a characteristic of the cell-based test system measured using the reference user equipment, wherein the information about the characteristic is stored at the server;prior to performing the test case on the user equipment to be tested, receiving the information about the characteristic from the server over the network; andsending, to the server over the network, results from performing the test case, wherein the shielded box is configured to contain the user equipment to be tested when performing the test case on the user equipment.

14. The system of claim 13, wherein the computer system is further operable for receiving software executable by the computer system from the server over the network.

15. The system of claim 13, further comprising a local test user equipment interface component comprising a base station coupled to the shielded box.