Patent Application
20120124428
The methods and systems described below relate to the field of software testing and development, and more particularly to methods and systems for simulating, testing and debugging of software in programmable devices with processors configured with embedded trace capabilities.
Testing software components is an important aspect of designing and developing programmable mobile electronic devices, such as cellular phones. During the process of designing and developing programmable devices, many tests are executed on each device to verify device characteristics, functionality, performance, security, error handling capability etc. Once the programmable device is released, quality assurance tests are routinely conducted to improve device performance, respond to customer feedback, and to ensure customer satisfaction. Also, there is often a need to identify the source of problems that are experienced in the field by mobile devices after they have been deployed so that corrective measures can be developed and deployed.
In order to properly execute such tests and diagnostic investigations, quality assurance specialists and end users typically execute the various features offered by the programmable device and report unexpected behavior and/or bugs. Software test engineers may develop use cases replicating the environment and processes that led to the unexpected behavior and/or bugs. Each use case may be executed on hardware replicating the programmable device and trace data may be collected from the execution.
Developing such use-cases is a time-intensive process and requires the configuration of numerous parameters. Also, each use-case execution generates a large amount of trace data, which must be analyzed to identify the source of the non-conformity, unexpected behavior and/or bug. Once the source of the problem is identified, the use case data and trace data are discarded.
Various embodiments are presented for systems and methods that provide for efficient troubleshooting and debugging of programmable device systems having processors configured with embedded instruction and data trace capabilities (referred to herein as embedded trace). In one embodiment, a use case may be derived based on an identified performance issue and/or bug. Once a use case is derived, the programmable device may be connected to embedded trace hardware and the use case duplicated on the embedded trace station. The use case may then be executed and the embedded trace data captured. Along with this embedded trace, relevant instruction flow data, state information and timing information may be extracted. This information may be then used to generate embedded trace data packages including at least one of: a program flow trace; a task modules list; software meta information; hardware meta information; use case specification; and a data flow trace.
Once the embedded trace data packages are generated, the embedded trace data packages may be inserted into a performance database to allow non-standard performance metrics to be extracted in later stages. Then, standard performance metrics (performance data) may be extracted from the embedded trace data packages. The performance data may be filtered, categorized, supplemented with metadata, indexed and stored in a database. After the performance data is stored, an intelligent comparison algorithm (herein smart diff) may be executed to generate comparison data between the newly stored data and previously stored performance data. The comparison data may be extracted from the smart diff tool and stored in an enterprise database.
In another embodiment, the values of data variables may be recorded at designated breakpoints and used to assist in the generation of the embedded trace data packages.
In another embodiment, the smart diff tool may be used to compare differences in predefined software logic blocks (herein functions) that are shared across multiple executions. Smart diff may be used these differences to generate information to be stored in the enterprise database. In yet another embodiment, smart diff may be used to analyze the impact of functions not present in previous executions on a particular use case. In yet another embodiment, smart diff may be used to analyze the impact of functions present in previous executions, yet not present in the current execution.
In another embodiment, the results of the smart diff comparison may be indexed and stored in an enterprise database such that this data may be accessed via a client application.
In another embodiment, smart diff may be used to analyze ETM trace packages across the time dimension and store the results in the enterprise database.
In another embodiment, smart diff may be used to analyze embedded trace data packages across the chipset dimension and store the results in the enterprise database.
In another embodiment, smart diff may be used to analyze embedded trace data packages across the use case dimension and store the results in the enterprise database.
In another embodiment, the client application may present users with an advanced search interface, allowing the enterprise database to be searched across the time dimension, the chipset dimension, and the use case dimension.
The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate exemplary aspects of the invention, and, together with the general description given above and the detailed description given below, serve to explain features of the invention.
The various aspects will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations are for illustrative purposes, and are not intended to limit the scope of the invention or the claims.
As used herein, the terms “programmable device” and “mobile device” are used interchangeably and refer to any one of various cellular telephones, personal mobile television receivers, tablet computers, personal data assistants (PDA's), palm-top computers, laptop computers, wireless electronic mail receivers (e.g., the Blackberry® and Treo® devices), and Internet enabled cellular telephones (e.g., the Blackberry Storm®), and similar personal electronic devices. A mobile device may include a programmable processor and memory as described more fully below with reference to
The testing and debugging of mobile devices typically involves creating use cases, or highly detailed and exhaustive test cases, which involve a time-intensive process that requires the configuration of numerous parameters. Executing a use case generates a large amount of trace data for analysis in order to identify the source of an identified non-conformity, unexpected behavior and/or bug.
Currently, use case data and trace data are discarded once the source of a problem is identified. It would be advantageous to provide a system and method that allows for the reuse of use cases and trace data. This is especially true for troubleshooting and debugging of mobile device systems implemented with processors configured with embedded trace capabilities, such as embedded trace macrocells (ETMs).
As used herein, the term “embedded trace” refers to any of a variety of technologies and software/hardware systems that enable real-time or near real-time collection of instruction execution and data transfers for use in development and debugging of embedded systems and software. A variety of such capabilities are available, including capabilities for real-time tracking of processor operations, instruction execution, bus transactions, and memory accesses. As used herein “embedded trace macrocell” and “ETM” refer generally to such embedded instruction/data transaction tracking capabilities, one example of which is the CoreSight™ Embedded Trace Macrocell™ technology licensed by ARM Ltd. However, references to “embedded trace macrocell” and “ETM” are not intended to limit the scope of the claims to a particular embedded trace technology or product.
ETMs are logic gates embedded into select modern processors, such as those made or licensed by ARM Ltd., that provide built-in debug and trace facilities. Specifically, ETMs collect instruction and data transfer information, compress this information and deliver the information, in real-time, to an external device for post processing. That is, ETMs send information regarding a processor's execution (for example, step-by-step functioning in the processor) to a separate device specifically designed to communicate with the ETM and collect the data. This allows the processor's information to be captured for use in testing and trouble shooting, while adding little or no burden to the processor's performance.
However, debugging software using ETMs is somewhat cumbersome and problematic. For instance, using an ETM requires expensive hardware, extensive training, and time-consuming use-case data generation. Further, ETMs generate a lot of trace information, which can be cumbersome to store, process and analyze. This is partially due to the fact that ETMs generate many gigabytes of trace data—the vast majority of which is simply disregarded or discarded. One reason the embedded trace data is discarded is that there is no facility for storing and retrieving such data.
The various embodiments overcome limitations of prior practice by providing for systematic storage and reuse of ETM trace data and other data generated in the process of testing and trouble shooting mobile devices. In the various embodiments systems and methods provide for efficient troubleshooting and debugging of programmable device systems having processors built with ETMs. A use case may be derived based on an identified performance issue and/or bug. Using the derived use case, the use case may be duplicated with the programmable device connected to ETM hardware and an ETM station. The resulting ETM trace data may be captured, from which relevant instruction flow data, state information and timing information may be extracted. This information may be used to generate ETM trace data packages such as program flow traces, task modules lists, software meta information, hardware meta information, use case specifications, and data flow traces. Performance data may be extracted from the ETM trace packages, and the performance data may be filtered, categorized, supplemented with metadata, indexed and stored in a database. Stored performance data may be analyzed by an intelligent comparison algorithm (which is referred to herein as “smart diff”) to generate comparison data contrasting newly stored data with previously stored performance data. Such comparison data may be extracted from the smart diff tool and stored in an enterprise database. The values of data variables may be recorded at designated breakpoints and used to assist in the generation of the ETM trace packages. The smart diff tool may be used to compare differences in predefined software logic blocks or functions that are shared across multiple executions. Smart diff may use these differences to generate information to be stored in the enterprise database. Smart diff may also be used to analyze the impact of functions not present in previous executions on a particular use case or versions of a mobile device being tested. Smart diff may also be used to analyze the impact of functions present in previous executions, yet not present in the current execution. In an embodiment, the results of the smart diff comparison may be indexed and stored in the enterprise database in a manner that enables this data to be accessed via a client application. With data stored in an enterprise database, smart diff may be used to analyze ETM trace data packages across the time dimension.
Currently, there are tools that can be used to process generated trace information and display the results of the trace information in a visual format.
An embodiment method 100 for reusing test and use-case data to generate a trace data is illustrated in
Once a use case is derived in step 102, a programmable device may be connected to an ETM device in step 105. This may be achieved, for example, by opening up communication lines between the programmable device and the ETM device using software network connection utilities. With the connection established between the programmable device and the ETM, the use case may be duplicated and transferred onto the ETM station in step 110. In step 115, the use-case may be executed on the programmable device and the ETM traces are captured and stored in memory of the ETM station.
In an embodiment, once the ETM traces are captured and stored in memory, instruction flow data may be extracted in step125. This may be accomplished by analyzing the trace data to identify and count or copy those traces associated with instruction branching operations. In step 130, state information may be extracted from the trace data, such as by selecting and recording individual state data. In step 135, timing information 135 may be extracted from the ETM traces, such as by noting the recorded time associated with particular operations or by noting a time difference between the start of one operation and the beginning of the next. In an embodiment, the values of data variables may be captured at pre-designated breakpoints in step 140. In another embodiment, the extraction of instruction flow data (step 125), state information (step 130), and timing information (step 135) may be extracted during the execution of the use case at the same time that the ETM traces are captured.
After the ETM trace information has been extracted and captured, this information may be used along with the captured instruction flow data captured in step 125, state information capture in step 130, and timing information captured in step 135 to generate ETM trace data packages in step 145. These ETM trace data packages may then be stored in a performance database in step 150. Storing the entire content of the ETM trace data packages allows non-standard performance metrics to be extracted in later stages. In an embodiment, the ETM trace data packages generated in step 145 may be stored in a database in step170. Any suitable database may be used for such purposes. In an embodiment, pre-configured processes may be executed on the ETM trace data packages in such manner as to extract relevant performance metrics (performance data) in step155. This performance data may then be filtered in step160, and embedded with metadata in step 165 before being stored in the database in step170. Such metadata may serve as an index or search terms for cataloging the information archived within the performance data and the original ETM trace data packages stored in the database. The performance data may also be organized and categorized as part of storing it in the database in step170.
Once the ETM trace data packages and/or performance data have been stored in the database, an intelligent comparator program (smart diff), may be executed on the newly entered database information in step175. Specifically, smart diff may be executed to compare the newly inserted information with previously stored information. Data similarities and differences may be analyzed in this process so as to determine how the use case has resulted in different performance, as well has how portions of the operations remain the same as in other or prior use cases. Also, a technical description of the difference between the data sets may be generated, (herein referred to as a differential comparison). This differential comparison may be categorized, analyzed, and stored in an enterprise database in step180. This process may be repeated for many use cases in order to generate a database of data across a range of uses and operational circumstances.
Data components making up an ETM trace data package 200 are illustrated in
An embodiment method 300 for using a client application to access the enterprise database is illustrated in
Referring to
The smart diff program may be further configured to analyze the ETM trace packages, or the extracted performance data, across a time dimension in module 520. Similarly, smart diff may be configured to analyze the ETM trace data packages and extracted performance data across a chipset dimension in module 525. Additionally, smart diff may be configured to analyze the ETM trace data packages and extracted performance data across a use case dimension in module 525. Further, for each of these dimensions of analysis, smart diff may be configured to evaluate one or more executions or use cases against other individual executions or use cases, a selected number of executions or use cases, or against all executions and use cases for which data is stored in the database.
In an embodiment, the programmable device 600 may further include a touch sensor 608 separate from the touchscreen display 610, which is coupled to the processor 601, such as via a touch sensor processing circuit 609, and positioned on or near a case of the mobile device 600. Such a touch sensor 608 may be a differential capacitance sensor, differential inductance sensor, thermal sensor, pressure sensor, ultrasound vibrations sensor, or any other sensor capable of detecting the touch of a user's hand and/or fingers to the case. Such a touch sensor 608 may be positioned on the sides and back of the mobile device 600 case, as well as long the bezel surrounding the touchscreen display 610.
The mobile device processor 601 may be any programmable microprocessor, microcomputer or multiple processor chip or chips that can be configured by software instructions (applications) to perform a variety of functions, including the functions of the various aspects described herein. In some mobile devices, multiple processors 601 may be provided, such as one processor dedicated to wireless communication functions and one processor dedicated to running other applications. Typically, software applications may be stored in the internal memory 602 before they are accessed and loaded into the processor 601. In some mobile devices, additional memory chips (e.g., a Secure Data (SD) card) may be plugged into the mobile device 600 and coupled to the processor 601. In many mobile devices, the internal memory 602 may be a volatile or nonvolatile memory, such as flash memory, or a mixture of both. For the purposes of this description, a general reference to memory refers to all memory accessible by the processor 601, including internal memory 602, removable memory plugged into the mobile device, and memory within the processor 601 itself.
The embodiments described above may also be implemented on any of a variety of computers, such as a workstation computer 60 illustrated in
The various embodiments may be implemented by a computer processor 61 executing software instructions configured to implement one or more of the described methods. Such software instructions may be stored in memory 62, 63 as separate applications, or as compiled software implementing an embodiment method. Reference database may be stored within internal memory 62, in hard disc memory 64, on tangible storage medium or on servers accessible via a network (not shown). Further, the software instructions and databases may be stored on any form of tangible processor-readable memory, including: a random access memory 62, hard disc memory 63, a floppy disc (readable in a floppy disc drive 64), a compact disc (readable in a CD drive 65), electrically erasable/programmable read only memory (EEPROM), read only memory (such as FLASH memory), and/or a memory module (not shown) plugged into the computer 60, such as an external memory chip or a USB-connectable external memory (e.g., a “flash drive”) plugged into a USB network port 66.
The embodiments described above may also be implemented on any of a variety of client-server systems, as illustrated in
The foregoing method descriptions and process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the steps of the various aspects must be performed in the order presented. As will be appreciated by one of skill in the art the order of steps in the foregoing aspects may be performed in any order. Further, references to step identifiers and words such as “thereafter,” “then,” “next,” etc. are not intended to limit the order of the steps; such identifiers and words are simply used to guide the reader through the description of the methods.
The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry that is specific to a given function.
In one or more exemplary aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module executed which may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to carry or store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a machine readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
The preceding description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
