Patent Application
20150138162
This application includes material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office files or records, but otherwise reserves all copyright rights whatsoever.
The disclosed system and method relate in general to the field of user interfaces, and in particular to user input testing systems.
The foregoing and other objects, features, and advantages of the disclosure will be apparent from the following more particular description of embodiments as illustrated in the accompanying drawings, in which reference characters refer to the same parts throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating principles of the disclosed embodiments.
This application relates to measuring or testing latency in user interfaces, including but not limited to fast multi-touch sensors of the types disclosed in U.S. patent application Ser. No. 13/841,436 filed Mar. 15, 2013 entitled “Low-Latency Touch Sensitive Device,” U.S. Patent Application No. 61/928,069 filed Jan. 16, 2014 entitled “Fast Multi-Touch Update Rate Throttling,” U.S. patent application Ser. No. 14/046,819 filed Oct. 4, 2013 entitled “Hybrid Systems And Methods For Low-Latency User Input Processing And Feedback,” U.S. Patent Application No. 61/798,948 filed Mar. 15, 2013 entitled “Fast Multi-Touch Stylus,” U.S. Patent Application No. 61/799,035 filed Mar. 15, 2013 entitled “Fast Multi-Touch Sensor With User-Identification Techniques,” U.S. Patent Application No. 61/798,828 filed Mar. 15, 2013 entitled “Fast Multi-Touch Noise Reduction,” U.S. Patent Application No. 61/798,708 filed Mar. 15, 2013 entitled “Active Optical Stylus,” U.S. Patent Application No. 61/710,256 filed Oct. 5, 2012 entitled “Hybrid Systems And Methods For Low-Latency User Input Processing And Feedback,” U.S. Patent Application No. 61/845,892 filed Jul. 12, 2013 entitled “Fast Multi-Touch Post Processing,” U.S. Patent Application No. 61/845,879 filed Jul. 12, 2013 entitled “Reducing Control Response Latency With Defined Cross-Control Behavior,” U.S. Patent Application No. 61/879,245 filed Sep. 18, 2013 entitled “Systems And Methods For Providing Response To User Input Using Information About State Changes And Predicting Future User Input,” U.S. Patent Application No. 61/880,887 filed Sep. 21, 2013 entitled “Systems And Methods For Providing Response To User Input Using Information About State Changes And Predicting Future User Input,” U.S. patent application No. 14/046,823 filed Oct. 4, 2013 entitled “Hybrid Systems And Methods For Low-Latency User Input Processing And Feedback,” U.S. patent application Ser. No. 14/069,609 filed Nov. 1, 2013 entitled “Fast Multi-Touch Post Processing,” U.S. Patent Application No. 61/887,615 filed Oct. 7, 2013 entitled “Touch And Stylus Latency Testing Apparatus,” and U.S. Patent Application No. 61/930,159 filed Jan. 22, 2014 entitled “Dynamic Assignment Of Possible Channels In A Touch Sensor,” and U.S. Patent Application No. 61/932,047 filed Jan. 27, 2014 entitled “Decimation Strategies For Input Event Processing.” The entire disclosures of those applications are incorporated herein by this reference.
In various embodiments, the present disclosure is directed to systems and methods to measure the input latency of direct-manipulation user interfaces on touch-sensitive and stylus-sensitive devices, such as touchscreens. Direct physical manipulation of digital objects/controls crafted to mimic familiar, real-world interactions is a common user-interface metaphor employed for many types of input devices, such as those enabling direct-touch input, stylus input, in-air gesture input, as well as indirect input devices, including mice, trackpads, pen tablets, etc.
For the purposes of the present disclosure, touch-input latency in a user interface refers to the time it takes for the user to be presented with a graphical, audial, and/or vibrotactile response to a physical touch or stylus input action. Tests have shown that users prefer low touch-input latencies and can reliably perceive end-to-end, graphical input-to-response latencies as low as 5-10 ms.
In various embodiments, touch-input latency in a single-touch, multi-touch and/or stylus user input device, and in the system processing this input, can have many sources. Such sources include, e.g., (1) the physical sensor that captures touch events, (2) the software that processes touch events and generates output for conveying to the user, (3) the output (e.g., display) itself, (4) data transmission between components, including bus, (5) data internal storage in either memory stores or short buffers, (6) interrupts and competition for system resources, (7) other sources of circuitry can introduce latency, (8) physical restrictions, such as the speed of light, and its repercussions in circuitry architecture, and (9) mechanical restrictions, such as the time required for a resistive touch sensor to bend back to its ‘neutral’ state.
While reducing touch-input latency is increasingly an important goal for many smart device manufacturers including but not limited to smartphone, tablet, PC, gaming, virtual reality, augmented reality and peripheral equipment manufacturers, at present there exists no standardized, inexpensive, reliable and repeatable system and/or method to measure and compare the touch-input latencies of smart devices during and after their manufacture.
In an embodiment, the presently disclosed system fills such unmet industry need by providing a means to measure the touch-input latency of a system at various points in the sensing/processing/display chain, including (but not limited to) the end-to-end latency of direct-manipulation user interfaces on smart devices, such as smart phones, tablets or computers, through the advent of a standardized, cheap, reliable and repeatable latency testing apparatus and method.
In an embodiment, the presently disclosed system measures the input latency of a touchscreen device. The term “input latency” herein means the difference in time between a touch event on a computing system's touchscreen and the touchscreen's resulting graphical, audial, or vibrotactile response to that touch event. As used herein, the term “touch event” and the word “touch” when used as nouns include a near touch and a near touch event, or any other gesture that can be identified using a sensor. For example, in a typical touchscreen application that includes digital button user-interface controls, a user will press a digital button with their finger, and the display will change to indicate that a digital button was pressed, such as by changing the color or appearance of that digital button user-interface control. The time between the user's touching of this digital button and the resulting graphical change is the input latency of a computing system, and the presently disclosed system can measure such touch-input latency.
In an embodiment, the latency measuring system 102 is configured to be intentionally dropped by a user onto a touchscreen 104 of the test device 103 whose end-to-end latency is to be measured.
In various embodiments, the latency measuring system 102 uses two sensors to measure the time at which the latency measuring system 102 physically contacts a touchscreen 104 and the time at which the resulting graphical change occurs in the touchscreen display in response to the hammer's 102 simulated touch input. The first sensor 204 can be a contact sensor that measures physical contact or near physical contact between the latency measuring system 102 and a touchscreen 104. The second sensor 205 can be a graphical-change sensor that measures the change in graphics displayed on a touchscreen 104 in response to the latency measuring system's simulated touch input.
In an embodiment, the latency of both of these sensors together is lower than the touch-input latency of the test device 103 that is being measured.
In other embodiments, the component of the overall time/latency measured which is due to the latency of the hammer's 102 internal components (sensors, circuitry, etc.) is fixed and subtracted from the measured end-to-end latency of a test device 103 when reporting that test device's touch-input latency.
In other embodiments, this hammer-based latency is variable, but detectable and detected, so that such latency or an estimate thereof can be subtracted from the reported end-to-end touch-input latency of a test device 103 when producing a reported latency measurement.
In an embodiment, the first sensor 204 (e.g., a contact sensor) can be a transducer that converts mechanical shock into an electrical signal.
In an embodiment, this transducer is a piezo-electric transducer.
In another embodiment, this transducer is a ferroelectric transducer. In another embodiment, this transducer is an accelerometer.
The first sensor 204 (e.g., a contact sensor) is positioned such that it converts the mechanical shock that occurs when the latency measuring system 102 contacts the touchscreen 104 into an electrical signal. The first sensor 204 is connected 208 to a computational engine 201 that is capable of measuring the changes in electrical signals. In an embodiment, the computational engine marks a first time when the first sensor is triggered and a second time when the second sensor is triggered, and determine the difference between the two times in connection with calculating latency. In an embodiment, the computational engine comprises a free running clock that is triggered upon a signal from the first sensor and stopped upon a signal from the second sensor, thereby providing data reflecting the measured latency.
In an embodiment, this change in signal emitted from the first sensor 204 is converted from an analog signal into a digital signal before it is measured by the computational engine 201.
The latency measuring system's computational engine 201 could comprise but is not limited to interconnected, processor and memory computer system components including system-on-a-chip (SoC), central-processing unit (CPU), micro-controller-unit (MCU), field-programmable gate-array (FPGA), NAND memory, vertical NAND (vNAND) memory, random-access memory (RAM), resistive random-access memory (ReRAM) and magnetoresistive random-access memory (MRAM).
In an embodiment, the second sensor 205 (e.g., a graphical-change sensor) is a photodetector that converts light into an electrical signal.
In an embodiment, the photodetector is a reverse-biased LED that acts as a photodiode. In various embodiments, this photodetector may be an Active Pixel Sensor (APS), a Charge-Coupled Device (CCD), a photoresistor, a photovoltaic cell, a photodiode, photomultiplier, phototube or quantum-dot photoconductor.
In an embodiment that might be useful for a reflective display technology, the second sensor 205 is paired with a light source that illuminates the display area to be sensed. The light source could be ambient light or could be a light source that is part of the latency measuring device.
The second sensor 205 (e.g., a graphical-change sensor) is positioned such that changes to the display result in changes in an electrical signal. The second sensor 205 is operably connected to the computational engine 201 that is capable of measuring the changes in electrical signals.
In an embodiment, this change in signal emitted from the second sensor 205 is converted from an analog signal into a digital signal before it is measured by the computational engine 201.
In an embodiment, the first sensor 204 (e.g., a contact sensor) and the second sensor 205 (e.g., a graphical-change sensor) are proximate such that contact between the latency measuring system 102 and a touchscreen 104 at the location of the first sensor 204 generates a change in the touchscreen's 104 displayed graphics that is sensed by the second sensor 205.
In an embodiment, the first sensor 204 and second sensor 205 are both housed in a common housing 206. This common housing 206 is connected to a second housing 203 that holds the computational engine 201. The second housing 203 can be shaped to be operable as a handle and the system 102 is configured to allow a user to hold the second housing and cause first housing to tap on an area of the device under test 103.
In another embodiment, the latency measuring system's computational engine 201 and sensors 204, 205 are placed in the same housing.
In an embodiment, the latency measuring system 102 includes an output 202 connected to the computational engine 201. The output 202 may comprise, e.g., a read-out display (e.g., an LCD, LED, OLED or electrophoretic display), a speaker, a light, a light reflective indicator, or a source of haptic output. After calculating the latency of the test device 103, the computational engine 201 sends this measurement to the latency measuring system's output 202 which then displays or otherwise identifies (e.g., audibly) the measurement or information reflective thereof to a user. The output 202 may provide an indication of a measurement of latency that is either qualitative or quantitative. The output 202 may be connected to the computational engine 201 by a wired or wireless interface.
In an embodiment, the output 202 is embedded within or mounted on the latency measuring system's chassis.
In an embodiment, the output 202 is discrete/unattached from the latency measuring system 102.
In an embodiment, the output 202 is the display of a discrete smartphone, tablet or PC including but not limited to the test device 103 under examination.
With reference to
With reference to
With continued reference to
In an embodiment, the discrete smartphone, tablet or PC used to perform and display the results of latency measurement from the readings of the latency measuring system's sensors is the test device 103 undergoing measurement.
In some embodiments, the mere contact between the latency measuring system 102 and the touchscreen 104 simulates a single, physical touch-down event from a user.
In an embodiment, the latency measuring system 102 includes a contact area 207 that rests between the sensors 204, 205 and the touchscreen 104. This contact area 207 is composed of a material having properties such that physical contact between the contact area 207 and the touchscreen 104 will produce touch input. Such material may comprise, e.g., any material known for incorporation into the tip of a stylus to cause the stylus to produce a touch input on a touchscreen.
In other embodiments, the contact area 207 of the latency measuring system 102 is designed, and the test device 103 is configured so that a physical touch-down event of the latency measuring system's contact area 207 will not be detected by the touchscreen 104 of a test device 103 until a signal is sent to it. This allows the user to rest the latency measuring system 102 on the display during a latency testing procedure.
In an embodiment, a computational engine 201 or other mechanism can generate a sequence of simulated single and/or multiple touch events (random, predetermined, or responsive to output) even when the contact area 207 of the latency measuring system 102 is placed in stationary physical contact with the touchscreen 104.
In an embodiment, these single and/or multiple touch events are simulated by injecting from the contact area 207 to the touchscreen 104 a signal that mimics signals that the touchscreen 104 controller of the test device 103 would normally interpret as a physical touch event.
In an embodiment, these signals are electrical and emitted from the latency measuring system's contact area 207 by one or more electrical signal generators 209 spaced apart wider than the row/column pitch of a capacitive touchscreen 104. In an embodiment, these electrical signal generators 209 are placed beneath a thin non-conductive contact area 207. In an embodiment, these electrical signal generators 209 protrude through the contact area 207 to contact the touchscreen 104 directly. In an embodiment, these electrical signal generators 209 are arranged in a grid. In an embodiment, these electrical signal generators are arranged in a random pattern.
Further embodiments of the system enable stationary touch-event simulation from the latency measuring system 102. In such embodiments, portions of the contact area 207 can be formed from dielectric materials whose dielectric characteristics are capable of being altered through electrical signalling controlled by the computational engine 201. These materials can comprise but are not limited to strontium titanate and barium strontium titanate. A touch event is mimicked by increasing the dielectric of the material so that, to the touch sensor, it appears that a finger, stylus or other dielectric object has come into contact with the touch sensor.
In another embodiment that enables stationary touch-event simulation from the latency measuring system 102, a contact area 207 on the latency measuring system 102 comprises one or more discrete pieces of dielectric or conductive material that are brought into physical contact with the touchscreen 104 using one or more linear actuators. The linear actuators and pieces of dielectric or conductive material are controlled by signaling from a computational engine 201. These materials could be but are not limited to plastic, rubber, water-filled containers, metals, metal-loaded plastics, etc. These linear actuators could be but are not limited to mechanical actuators, hydraulic actuators, pneumatic actuators, fluidic systems, piezoelectric actuators, electro-mechanical actuators, linear motors or rotary-driven cams. These computationally-controlled signals could be but are not limited to electrical, optical, mechanical, pneumatic, hydraulic and acoustic signals.
In an embodiment, the touchscreen 104 and not the latency measuring system 102 is connected to one or more linear actuators which moves it into physical contact with the latency measuring system's contact area 207 to generate simulated touch-tap and touch-drag events without human user intervention.
In an embodiment, the contact area 207 is deformable and mimics the deformation of a human finger when a finger comes into physical contact with a touchscreen 104. In an embodiment, the degree of deformation is measured and recorded. In another embodiment, the degree of deformation at a given time is used in performing the touch-latency measurement calculations herein described. In particular, the degree of deformation could be used in place of or alongside the first sensor 204 (e.g., a contact sensor) to indicate the moment of physical contact between the contact area 207 of the latency measuring system 102 and the touchscreen 104 of a test device 103.
In an embodiment, the contact area 207 is translucent to light such that the second sensor 205 is capable of sensing graphical changes through the contact area 207.
In another embodiment, the contact area 207 includes physical openings or holes such that the second sensor 205 can sense graphical changes emitted by a test device 103 in response to touch input through one or more of these openings or holes. These opening or holes in the contact area 207 can either be covered by a substrate translucent to light, or be left completely uncovered.
In an embodiment, the contact area 207 is composed of materials such that the first sensor 204 can sense the transferred shock between the touchscreen 104 and contact area 207.
In an embodiment, the test device 103 is configured to run software that causes its touchscreen 104 to graphically respond to simulated touch inputs from the latency measuring system 102 in a manner such that those graphical responses can be easily measured and recorded by the second sensor 205. For example, in an embodiment, a software program could be pre-installed on the test device 103 so that a touchscreen 104 remains entirely black when no physical touch-input events are received and then changes to entirely white when a single touch-input event is received.
In another embodiment, a pre-installed software program on the test device 103 could set a touchscreen 104 to remain entirely black when no physical touch-input events are received, and then to render white squares in response to each simulated touch event received from the latency measuring system 102 where each of the white-square graphical responses are sized appropriately to ensure easy detection by the second sensor 205 (e.g., a graphical-change sensor).
In another embodiment, a pre-installed software program on the test device 103 could provide a user with visual on-screen guides and instructions about where, when and how to bring the latency measuring system 102 into physical contact with the touchscreen 104 of a given test device 103 to ensure uniformity of measurement methodology across multiple latency tests and/or multiple test devices 103. For example, in an embodiment, a pre-installed software program on a test device 103 could draw an empty white box on its touchscreen 104 alongside on-screen textual cues to indicate at which point on the touchscreen's 104 surface a user should bring the latency measuring system 102 and the touchscreen 104 into physical contact. In another embodiment, a pre-installed software program on a test device 104 could draw a linear path with arrows and textual annotations indicating exactly where and at what velocity a user should drag a latency measuring system's contact area 207 across a touchscreen 104. In another embodiment, a pre-installed software program on a test device 104 could show a user a numeric countdown in seconds to inform him or her exactly how long the latency measuring system's contact area 207 should rest on a touchscreen 104 before lifting it off of the test device 104.
In an embodiment, the computational engine 201 is configured to measure and identify key features in the signal represented by the line graph 301, which may include one or more of the following—a rapid rise (or fall) in the signal 302, a peak in the signal 303, a valley in the signal 304, or the crossing of a signal threshold. The time at which this measured feature occurs is recorded by the computational engine 201.
As indicated above, the second sensor 205 can be a photodetector that converts light into an electrical signal and the second sensor 205 is connected 208 to a computational engine 201 that can sense the change in this signal over time.
In an embodiment, the computational engine 201 is configured to identify key features in this line graph 401, which may include one or more of the following - a rapid rise (or fall) in the signal 402, a peak in the signal 403, a valley in the signal, or the crossing of a signal threshold. The time at which this feature occurs is recorded by the computational engine 201.
In an embodiment, the standoff 702 is a linear actuator. In variations of this embodiment, the standoff 702 could be composed of but is not limited to a mechanical actuator, a hydraulic actuator, a pneumatic actuator, a fluidic system, a piezoelectric actuator, an electro-mechanical actuator, a linear motor or a rotary driven cam.
In an embodiment, the bottom of the latency measuring system 102 is a contact area 207 comprised of one or more second sensors 205 (e.g., graphical-change sensors), a touch-stimulation region 801 composed of materials that when in physical contact with a touchscreen 104 of a test device 103 mimics physical touch input events, and a first sensor 204 (e.g., a contact sensor) to detect when the latency measuring system 102 comes into physical contact with a test device's touchscreen 104.
In an embodiment, the touch-stimulation region 801 of the latency measuring system 102 is composed of a material with properties such that physical contact between the contact area 207 and the touchscreen 104 will produce touch input.
In other embodiments, the touch-stimulation region 801 of the latency measuring system 102 is designed, and the test device 103 is configured, so that a physical touch-down event of the touch-stimulation region 801 on a touchscreen 104 will not be detected by the touchscreen 104 of a test device 103 until a signal is sent to it. This allows the user to rest the latency measuring system 102 on the display during a latency testing procedure.
In an embodiment, the touch-stimulation region 801 is composed of a dielectric or conductive material.
In an embodiment, the touch-stimulation region 801 can be connected to a computational engine 201 to generate signals that simulate physical touch-down events on a touchscreen 104 as made by one or multiple fingers or styli when the latency measuring system 102 is stationary or moving.
As discussed in prior embodiments, when the latency measuring system 102 is stationary this enables the touch-stimulation region 801 to stimulate multiple, simultaneous and readily repeatable physical tap events on a touchscreen 104 of a test device 103 without requiring a human user to manipulate the latency measuring system 102.
By contrast, embodiments that simulate multiple physical touch events on the touch-stimulation region 801 while the latency measuring system 102 is being physically dragged by a user enable the system 102 to simulate complex multi-finger/styli gestures and ink events on a touchscreen 104 during a diagnostic procedure.
As aforementioned with reference to prior embodiments, such a computationally-controlled touch-stimulation region 801 could be composed of electrical signal generators 209 controlled by a computational engine 201 spaced apart wider than the row/column pitch of a capacitive touchscreen 104, dielectric materials whose dielectric characteristics can be altered through electrical signalling controlled by a computational engine 201, or the combination of one or more discrete pieces of dielectric or conductive material with one or more linear actuators.
In an embodiment, the second sensors 205, touch-stimulation region 801, first sensor 204, and output 202 are connected to a computational engine 201.
In another embodiment, only the second sensors 205, the first sensor 204, and output 202 are connected to a computational engine 201.
In another embodiment, only the second sensors 205 and the first sensor 204 are connected to a computational engine 201.
In an embodiment, the second sensors 205 are photodetectors that convert light into an electrical signal.
In an embodiment, the photodetectors are reverse-biased LEDs that acts as photodiodes. In various embodiments, this photodetector may be an APS, a CCD, a photoresistor, a photovoltaic cell, a photodiode, a photomultiplier, a phototube or quantum-dot photoconductor.
In an embodiment, the second sensors 205 are paired with one or more light sources that illuminate the touchscreen 104 area to be sensed by the second sensors 205. These light sources which improve the accuracy of second sensor 205 measurement across a variety of surfaces including glass, glossy or reflective surfaces could be but are not limited to one or more light-emitting diodes (LEDs) or infrared laser diodes.
The second sensors 205, which may be graphical change sensors, are positioned such that changes to a test device's display result in changes in an electrical signal, and such that they can detect recognizable features and patterns on the cover substrate of a test device's touchscreen 104. The second sensors 205 may be connected to a computational engine 201 that is capable of measuring the changes in electrical signals.
In an embodiment, this change in signal emitted from the second sensor 205 is converted from an analog signal into a digital signal before it is measured by the computational engine 201.
In an embodiment, the first sensor 204 and second sensor 205 are proximate such that contact between the latency measuring system 102 and a touchscreen 104 at the location of the first sensor 204 generates a change in the touchscreen's 104 displayed graphics that is sensed by the second sensor 205.
In an embodiment, the latency measuring system 102 is dragged around the touchscreen 104 of a test device 103 whose end-to-end drag latency is to be measured, the display of the touchscreen 104 shows a graphical marker 802 where the test device 103 believes the physical touch event mimicked by the latency measuring system 102 to be.
In an embodiment, the display of a test device's touchscreen 104 can be programmed to render a graphical marker 802 in any color, pattern and/or shape at any time and to display that graphical marker 802 against a colored and/or patterned background at any time that improves the accuracy of second sensor 205 measurement of both the position of a graphical marker 802 and of the position and velocity of the latency measuring system 102 at a given point in time.
In an embodiment, the position and velocity of the latency measuring system 102 as it is dragged across a touchscreen 104 is measured using an optical technique similar to that of an optical, laser, glaser, or dark-field mouse.
In an embodiment, the position and velocity of the latency measuring system 102 is measured using circuitry and an electro-mechanical measurement technique similar to that of a trackball mouse.
By measuring the velocity of the latency measuring system 102 across the touchscreen 104 and the physical distance between the center of a graphical marker 802 and the center of the system's physical touch-stimulation region 801, the latency measuring system 102 can measure the end-to-end touch-drag latency of a test device 103. If the velocity of the latency measuring system 102 is constant, the measured drag latency is equal to the physical distance between the center of a graphical marker 802 and the center of a system's touch-stimulation region 801, divided by the speed at which the latency measuring system 102 is travelling. If the velocity of the latency measuring system 102 is not constant, the relationship between the latency measuring system 102 and the relative position between the touch-stimulation region 801 and the graphical marker 802 will be a more complicated mathematical function.
In an embodiment, the position and velocity of the latency measuring system 102 are measured using the second sensors 205 and graphical changes and/or markers that are imposed on the display of a test device's touchscreen 104 for this purpose.
In an embodiment, the latency measuring system 102 combines the functions of tap and drag latency measurement through the addition of a first sensor 204 (e.g., a contact sensor) to the embodiment, so that it can measure the input latency of a test device's touchscreen 104 in either or both types of touch and/or stylus interactions. In this embodiment, as previously explained, the difference between the time of the latency measuring system's first physical “tap” contact simulation on the touchscreen 104 and the detection by the system's second sensors 205 of a pre-programmed graphical response to that simulated physical contact can be used to calculate the round-trip, touch or stylus input latency of a test device 103 for tap events.
In an embodiment, the system 102 is designed such that multiple latency measuring systems can be used simultaneously. In certain embodiments, the systems are wielded by multiple hands as they are placed on the touchscreen 104 of a test device 103. The positions and actions of the latency measuring systems 102 are disambiguated by the sensors they employ to determine position, velocity, tapping, etc.
As aforementioned, in an embodiment, the connections between the computational engine 201 and the latency measuring system's first sensor 204, second sensors 205 and the touch-stimulation region 801 could be physical wires or wireless including but not limited to bluetooth, bluetooth low-energy (BLE), WiFi or WAN.
Use of existing sensor components as well as the addition of new sensor components to the latency measuring system 102 connected to a computational engine 201 can enable a latency measuring system 102 to test the touch-input latency of other forms of a test device's response to touch-input beyond graphical responses, including audio and vibrotactile responses.
In an embodiment, the latency measuring system 102 can contain one or more first sensors 204 connected to a computational engine 201 to detect not only the first physical contact between the latency measuring system 102 and a test device's touchscreen 104, but also a test device's vibrotactile response. The difference in the latency measuring system's measured timing between the system's first simulated physical touch input event and a test device's first detected vibrotactile response is the touch-input latency of a test device's vibrotactile response to tap gestures. Similarly, the measured timing of changes in the frequency or intensity of a test device's vibrotactile response to multi-touch gesture, inking or drag commands recorded by the latency measuring system's first sensors 204 can be compared to the measured timing of when those finger/styli gestures were simulated by the latency measuring system 102 to determine the touch-input latency of a test device's vibrotactile response. As in computing the touch-input latency of graphical responses, any fixed or variable latency in the system's internal sensors and/or circuits used to detect a test device's response should be estimated and subtracted from the system's reported measurements of a test device's vibrotactile touch-input latency. As a latency measuring system's first sensors 204 can be connected to the system's computational engine 201 by a long physical wire or wirelessly, they can also be easily placed proximate to a test device's vibrotactile feedback components to ensure the most accurate readings of a test device's vibrotactile response latency.
As shown in
In various embodiments, the device being used to measure latency can be the same device whose latency is being measured. In some such embodiments, the “touch” can be initiated by a user, and effectively simultaneously detected by the touch sensor and by another sensor in the device, such as an accelerometer, magnetometer, microphone, proximity sensor, or any sensor known to one skilled in the art (or asynchronously recorded, and compensated for). The response whose time is subsequently measured might be visual, in which case a camera or other light sensor might be used, or it might be auditory, in which case a microphone or other audio sensor might be used. In other embodiments, the response of an internal system or application process may instead be recorded, without actually outputting to an output device. In some embodiments, a mirror or other reflective surface may be required to cause the display to be visible to the camera. In other embodiments, sensors might be positioned to enable direct observation of the output (or surface to reflect sound from the speakers). In some embodiments, the initial ‘touch’ might be simulated, through the generation of one or more false ‘touches’. This might be injected in software. It might also be generated through electrical or other trigger of the touch sensor itself. In some embodiments, some portion of the time between input and response might be estimated (eg: presumed to be some fixed time for a particular handset, class of device, sensor type, sensor model, etc). In other embodiments, the output of one or more sensors might be aggregated, so-as to provide an average response time (or other statistical means of aggregation).
Latency Testing Apparatus
Current Testing Methods
Simple techniques have been utilized for measuring the touch-to-display latency. The device is placed in a mode where a graphic is meant to follow a touch point. (e.g., inking, translating an icon). A ruler is placed on top of the device. A finger is then dragged across the screen along the ruler edge at as constant a rate as possible. A high speed camera of known frame rate captures the finger motion as well as the resulting graphical response. By examining individual frames in the video, the speed of the finger (distance travelled between successive frames×frame rate) can be calculated. The amount of separation between the finger and the following graphic can be directly observed. The latency can be calculated as the separation distance/the finger speed. Two measurements are typically taken, worst-case latency and best-case latency.
Novel Latency Apparatus and Methods
Throughout this disclosure, the term “input event” is used generically to describe the issuance of a command to an input device. It is not intended to be specifically limited to user interfaces which make use of event-driven software architectures.
In an embodiment, the device-under-test 103 is not held immobile, but its position and orientation with respect to an input means (such as a finger/stylus proxy device) is measured or well-known.
An input proxy device 1007 is provided, and may comprise a plurality of artificial touch or stylus user input devices that provide a finger/stylus proxy device. In an embodiment, the capacitive coupling between the input proxy device 1007 and the device-under-test 103 can be varied by electrical, mechanical, chemical or other means. Such finger/stylus proxy devices can be made of a plurality of materials to simulate differences of capacitance, contact area, and friction against the screen of the device-under-test.
At least one mechanical motor is provided for moving the input proxy device 1007 with respect to the test device 103 or moving the test device with respect to the proxy device. In an embodiment, at least two mechanical motors are provided in an X-Y table 1009 to allow for x, y, and z movement of the input proxy device 1007, as well as motion and orientation at a variety of angles to the device-under-test, as well as to change the velocity and pressure/force along all of these axes and degrees of freedom. In an embodiment, the input proxy device is held stationary and the device holder 1001 is mounted on an X-Y table having motors for providing X-Y motion of the device holder 1001.
A test application loaded onto the device-under-test 103 is provided for recording movements of the input proxy device 1007 and displaying a representation of this movement. In an embodiment, the test application displays a blank, white screen and the representation comprises a single black ink-stroke. In an embodiment, the test application has zero and high “SoC-load” simulation states to assess how strain on the CPU and GPU impacts touch/stylus performance. The high “SoC-load” case can be defined as a set of commonly-available/used, open, and backgrounded third-party applications that a typical smart-device user might multitask across in daily use. In an embodiment, the test application uses a plurality of “SoC-load” states corresponding to different CPU loads from zero load to the maximum CPU load as defined by the device-under-test. In this respect, the test application in combination with other elements of the invention can measure how well the device under test handles additional load. For example, the test application in combination with other elements of the invention can measure whether the latency becomes longer relatively slowly or relatively suddenly with additional load, i.e., measure a rate at which latency becomes longer with additional load. In this respect, more than two load states may be utilized by the system. The test application in combination with other elements of the invention can also separate out load on the CPU from bottlenecks in other areas, such as short and long term memory access. The test application can be configured to toggle between both a GPU-accelerated (in an embodiment accelerated by DirectX or OpenGL) and CPU-only black ink-stroke in response to finger/stylus proxy device behavior. These types of modes are commonly referred to as rendering in hardware and rendering in software, respectively.
The test application and other aspects of the system can utilize a standard third-party application configuration across all devices-under-test to be tested. In this respect, the application can be configured to account for different platforms of different devices under test having different versions of third-party applications, or even the “same” application on different platforms being coded differently. Load-levels may be based on calculations per second or a purely mathematical measure, not to replace “standard application load” measurement, but rather to complement it.
In an embodiment, at least one ground-truth measurement apparatus 1011 is utilized to record finger/stylus proxy device movement with respect to the device-under-test 103 as well as the black ink-stroke generated by the test application to provide a ground-truth. In an embodiment, the ground-truth measurement apparatus 1011 comprises a high-speed camera. In another embodiment, the ground-truth measurement apparatus 1011 comprises a one-, two-, or three-axis accelerometer attached to the finger/stylus proxy device. This accelerometer senses the acceleration of the finger/stylus proxy device over time, and this data is sent to and can be recorded by the testing application.
In a related embodiment, the device-under-test outputs a video signal, such as a VGA, HDMI, DP, DVI, LVDS, or other video signal, that is processed by an additional application. Through reading the video signal, the additional application can determine the shape and position of black ink strokes through looking at the timing and/or packet information in the video signal. In a related embodiment, the device-under-test outputs a video signal to another device or display, and the ground-truth measurement apparatus “taps” this connection to read the data passed over it without interfering with its intended operation.
In an embodiment, the system further includes a computing device and administrative application that generates a random but user-editable input proxy device movement sequence, controls the motors and thereby the movement sequence of the finger/stylus proxy device, recognizes/tracks the movement of the tip of the finger/stylus proxy device, and measures any divergence in space and time between the black ink-stroke of the test application and the actual, predefined movement sequence of the finger/stylus proxy device.
Numerous variations of the testing procedure and apparatus are possible. In an embodiment, the finger of the tester is used directly. In an embodiment, the user wears a glove or other covering on the touching member that can sense when a touch or other input event has occurred (with negligible or at least well calibrated latency). In an embodiment, the tester wears a finger extension or finger phantom. In an embodiment, a finger phantom is moved by mechanical means. In an embodiment, signals (electrical or other appropriate type) are induced at one or more points in the sensing, processing and display chain to simulate the effect of real input events at those points in the chain. In an embodiment, the touching device (finger, phantom, etc.) is stationary and the input device moves. In an embodiment, the system is configured such that both the touching device and the input device move in a predefined or at least well-measured manner.
Apparatus Assembly
In an embodiment, the apparatus is constructed with the device-under-test placed horizontally. The finger/stylus proxy device is attached to a mechanical “arm” that is mounted above the device-under-test. A plurality of mechanical motors are positioned so as to enable simultaneous control of the position, angles, pressure, and velocity of the finger/stylus proxy device in all desired dimensions.
In an embodiment, a ground-truth measurement apparatus sensor (which could be an optical-sensing means for determining the state of the device-under-test's display) is mounted at a fixed point above the entire apparatus to allow for a view at all times of the entire device-under-test placed on the device-tray, finger/stylus proxy device, and the test application's black ink-stroke-regardless of the degree of lag between the finger/stylus proxy device and the black ink-stroke.
In an embodiment, the GTM2 sensor is mounted directly parallel to the finger/stylus proxy device to allow for a close, occlusion-free view of any divergence between the movement of the finger/stylus proxy device and the test application's black ink-stroke.
In an embodiment, two GTM2 sensors are used: one mounted to a fixed point above the entire apparatus and one mounted directly parallel to the finger/stylus proxy device.
In an embodiment, one accelerometer-based GTM2 sensor is directly attached to the finger/stylus proxy device so as to record its acceleration. In an embodiment, a second accelerometer-based GTM2 sensor is attached to the device-under-test as to measure the relative acceleration between the device-under-test and the finger/stylus proxy.
In an embodiment, the device-under-test is connected to a GTM2 sensor through its video output port (VGA, DVI, DP, HDMI, etc.).
Testing Procedure
In an embodiment, software is loaded onto the device to be tested. This software contains predetermined visual responses to various input event types. In some embodiments, this software might include user interface components comprised by a user interface toolkit present on the device to be tested. In other embodiments, this software might generate all responses to input ‘from scratch’. In still other embodiments, both types of responses might be included.
In an embodiment, the software is a standardized test package, designed for comparing various versions, models and products.
In an embodiment, the software is an existing software package, whose latency and error performance we would like to characterize.
In an embodiment, once the software has been loaded onto the device to be tested, it is executed. The testing apparatus is then used to make various inputs to the device. Those inputs are recorded as ‘ground truth’ (in some embodiments, using techniques such as computer vision to track the position of the apparatus; in others, the position of the stylus/finger is tracked mechanically. In further embodiments, an accelerometer tracks the acceleration of the finger/stylus proxy rather than its position.). In this embodiment, the device generates responses to the input actions in accordance with the software installed on the device. Those responses are then monitored by the testing apparatus, and compared with the ‘ground truth’ of the input action to determine latency.
In an embodiment, the finger/stylus proxy device includes an active or passive marker to aid in tracking its position.
In an embodiment, the outputs of the device are visual, and sensed using optical sensing techniques. In an embodiment, the outputs are aural, and sensed using a microphone or other sound input device. In an embodiment, the outputs are physical, and are sensed using a vibration sensor or other positional sensing device. In an embodiment, the outputs are in taste or smell, and measured through chemical sensors or other taste/smell sensing methods.
In some embodiments, latency is measured as the duration of the interval between an input action and the time the response is generated. In some embodiments, the interval between responses is measured (this is often but not always the ‘frame rate’ of the output device). In some embodiments, the deviation of the nature of the output from the expected is measured. This includes, but is not limited to, the deviation of an output path from the ground truth, or the differences in intensity of output from that expected given the ground truth.
In an embodiment, the ground truth acceleration of the finger/stylus proxy (as measured by the GTM2) is compared against the path of input as recorded by the testing software. In this embodiment, latency is measured as the time difference between the recorded acceleration changes of the finger/stylus proxy and the associated changes in path of input as recorded by the testing software. In a variation of this embodiment, the time component of the acceleration data provided by the GTM2 is shifted to account for any known and constant latency in the accelerometer itself.
In an embodiment, the ground truth position of the finger/stylus proxy is measured through one of the means above, and the shape and position of the black ink stroke in the testing application is measured through a second GTM2 sensor attached to the device via a display cable. Latency is measured as the time difference between points in the stroke as measured by the two GTM2 devices. In a related embodiment, the GTM2 sensor attached to the device-under-test via a display cable is calibrated to account for any latency introduced by the packaging and sending of display information over the cable.
In an embodiment, the various measures of the apparatus are recorded, and made available as a data file. In an embodiment, a report is automatically generated. In an embodiment, a report is automatically generated, and placed into a datastore for subsequent retrieval. In an embodiment, the results are made available through a web interface.
Testing of Various Input Events
Various types of input event types are tested. In general, the apparatus will be expanded to include event types supported by the device being tested. This set of events generally includes input primitives that may be specifically called-out in the software of the device (eg: ‘tap’, ‘drag’, ‘rotate’, etc). This also includes events that are not specifically detected by the device software, but rather are interpretations of input by application software running on the device (eg: as a user drags their finger across the device, the system software may simply generate a series of ‘drag’ events, but the software monitoring those events would respond in a particular way to a particular drag pattern, such as drawing a character on the screen). In any event, all events of these or any other type may be included in the testing.
In general, any sensor included in the device may be tested. This includes touch sensor, accelerometer, gyroscope, camera, proximity sensor, 3D camera, hover-position sensor, keyboard, direct-touch sensor, and many others. Examples of the sorts of input events to be tested include the following:
Tapping
Time from when physical contact is made until various points in the S/P/D (Sensing-Processing-Display) chain.
Sliding, dragging
Differences in the time and spatial values of the various points in the S/P/D chain and the true value of the touching apparatus (finger, stylus, etc.)
Rotating
Difference in the time and spatial values of various points in the S/P/D chain from the expected positions given the ground truth.
Hover
How well the true position of an input event can be predicted or enhanced from information gathered by non-touch sensors.
Device Rotation
Difference in the time and rotational values of the device itself of various points in the S/P/D chain.
Device Position
Differences in the time and spatial values of the device itself various points in the S/P/D chain and the true value of the touching apparatus (finger, stylus, etc.)
Camera lag
Differences in the image shown-on screen from the expected content given the ground truth at various points in the S/P/D chain.
Proximity-sensor position
Differences in the time and spatial values of the device proxy at various points in the S/P/D chain and the true value of the touching apparatus (finger, stylus, etc.)
Evaluation of the Measured Latencies
The measured latencies can be reported as-is (time differences), or they can be reported as time differences along with the corresponding spatial errors, or they can be reported as a statistical function of the time differences and spatial errors.
The measured latencies can be compared with those of measurements taken on a plurality of input devices (different instances or versions of the same model, different models, etc.) and reported as either a qualitative or quantitative comparison between the devices.
The measured latencies can be compared in a user-subjective manner, with the report being a metric of the equivalent “user experience”.
The present system and methods are described above with reference to block diagrams and operational illustrations of methods and devices for measuring or testing latency. It is understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, may be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implements the functions/acts specified in the block diagrams or operational block or blocks. In some alternate implementations, the functions/acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
While the invention has been particularly shown and described with reference to a preferred embodiment thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
This application is a non-provisional of U.S. Application No. 61/887,615 filed 7 Oct. 2013. This Application is a continuation-in-part of U.S. application Ser. No. 14/316,177 filed 26 Jun. 2014. The entire disclosures of those applications are incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61887615 | Oct 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14316177 | Jun 2014 | US |
| Child | 14508916 | US |
