Patent Application
20180307330
The present invention applies to the field of Human-Computer Interaction (HCI). More specifically the focus of the invention is on measurement and calibration of end-to-end latency of a Man Machine Interface (MMI).
A MMI comprises one or more of an input interface and an output interface connected to a processing device. Input interfaces have evolved from a simple keyboard to a plurality of modalities including, by way of example, a 2D or a 3D mouse or pointer, a joystick, a touch pad or a touch screen, a microphone or a camera. Output interfaces comprise, among others, displays, video projectors, loudspeakers or headphones, signaling devices, etc. The processing devices with which a human may interact have also diversified: they now include, in addition to traditional desktop, laptop, server computers or TVs, game consoles, tablets, smart phones, but also robots, for example humanoid robots, and also aircrafts, cars or home appliances. An interaction may be local or at a distance, over a network, through a plurality of machines. An interaction may involve a single human or a plurality of humans or living beings.
The experience of a user when dealing with an application running on a processing device will depend to a high degree on the quality of the MMI. Quality depends not only on the ease of use but also on how reactive is the output interface to entries in the input interface. This parameter of quality is significantly affected by the time it takes for an input to be converted into an output by the processing chain. This time is called end-to-end latency. In local systems with one human only in a simple loop of a mouse command to a display of a cursor movement, latency may be as low as a few ms or as high a number of tens of ms. A latency of 50 ms is known to degrade the user experience. Also, variance of the latency (i.e. jitter) will be badly perceived.
This is particularly annoying for professional applications like air traffic control or aircraft piloting, but also for interaction intensive consumer applications like video games. Application developers have therefore tried to find ways to compensate latency and/or jitter. To do this it is necessary to measure these factors. This has been classically done for a mouse to display loop by using an external acquisition device such as a camera to detect the movement of a cursor on a screen, said movement being controlled by a mouse. The process requires a synchronization of the processing device and of the camera, which is not easy to achieve with sufficient precision.
The invention solves this problem of the prior art by providing an arrangement wherein the measurement of latency uses a single clock, shared by the processing device and the acquisition device.
To this effect, the invention discloses an arrangement configured to calibrate latency of a processing device between an input interface and an output interface, said arrangement comprising: a generator of a calibrated output to be output by the processing device through the output interface; a sensor configured to capture the calibrated output and send it to a processing unit of the processing device through the input interface.
Advantageously, the sensor is an optical sensor.
Advantageously, the input interface is a port of the computer device adapted to receive datasets from computer mice.
Advantageously, the calibrated output is a two-dimensional periodic image configured to move on a screen of the processing device relatively to the optical sensor.
Advantageously, the two-dimensional periodic image comprises elementary patterns, each elementary pattern comprising at least a number N of pixels of two different colors, at least n pixels in each elementary pattern being of a first color of a bright shade with a contrast to the other pixels of a second color equal to or higher than a preset threshold.
Advantageously, the number N of pixels is determined by taking into account a number of pixels of the optical sensor, a size of an image of the pixels of the optical sensor and a resolution of the screen.
Advantageously, n is higher than or equal to a minimum number of pixels of the first color which are needed by a tracking algorithm of the sensor and lower than or equal to a maximum number of pixels of the first color which define characteristic points at their boundary which remain invariant by a translation of one pixel of the elementary pattern in a direction of motion of the calibrated output.
Advantageously, N equals 9 and n equals 2.
Advantageously, at least two pixels in the n pixels of the first color are diagonally contiguous.
The invention also discloses the two-dimensional periodic image used in the arrangement of the invention or stored on a computer media.
The invention also discloses a processing device comprising an input interface and an output interface, said processing device being configured to operate a routine of compensation of a latency between an input command input at the input interface and an output output at the output interface based on execution of the input command by the processing device, said routine of compensation of a latency being configured to receive a set of parameters produced by a calibration routine, said calibration routine using results of an interaction between a sensor and a generator of a calibrated output at the output interface.
Advantageously, the calibration routine is first performed to produce a table of parameters and the routine of compensation is further configured to receive the set of parameters from a calculation based on the table of parameters.
Advantageously, the calculation based on a table of parameters uses settings of one or more of hardware and software settings of the processing device and the calibration routine and the routine of compensation are chained, so that compensation of latency is performed in real-time.
The invention also disclose a software development toolkit configured to develop interactive applications, comprising a modeler of a latency between input commands input at an input interface of a processing device and an output interface of the processing device, wherein said modeler of a latency is configured to perform one or more of measuring and compensating latency, measuring latency being based on said input interface being configured to receive an input generated by an interaction of a sensor with a generator of a calibrated output.
Advantageously, the software development toolkit of the software development toolkit of the invention is further configured to perform one or more of measuring and compensating latency based on a segmentation of said latency between one or more hardware and software segments of an environment of the processing device.
Thanks to the invention, a simple way is provided to measure, calibrate and compensate latency and jitter. A specific arrangement of the invention is one using an optical sensor. The arrangement of the invention can work for a number of other input interfaces and output interfaces, like audio interfaces. In some embodiments, the calibration allows real time compensation of latency and jitter.
The invention will be better understood and its various features and advantages will become apparent from the description of various embodiments and of the following appended figures:
The end-to-end latency (or lag) of a graphical user interface is commonly defined as the duration between a user action (e.g. movement of an input device or human limb) and the corresponding on-screen visual feedback (e.g. cursor movement, object displacement). All interactive systems have some latency introduced by input and output devices, network, events handling and processing. Latency is difficult to measure and is only occasionally reported on input or output devices datasheets, interactive systems specifications or MMI experiments reports. Latency and its variation over time, called latency jitter, have been known for a long time to affect human performance and qualitative perception. A latency of around 50 ms is known to affect performance in mouse-based pointing tasks and latency jitter above 20-40 ms is likely to be noticed. The effect is even more pronounced in direct touch interaction, where it has been shown that latency as low as 2 ms can be perceived and that performance is affected from 20 ms.
Credit for the photo on the left hand part of the figure is given to the publication from which it is extracted (“Designing for Low-Latency Direct-Touch Input”, Albert Ng et alii, UIST'12)
It is the most common procedure to measure end-to-end latency. An external camera 110 is used to record both the input device 120 (or finger) and the associated response on screen. An analysis of the video is then performed to match the two related events in order to count the number of elapsed frames 130 between them.
Different variants of equipments may be used to measure latency, for instance the ones disclosed by Teather, Pavlovych, Stuerzlinger and MacKenzie in “Effects of tracking technology, latency and spatial jitter on object movement”, Proceedings of 3DUI '09, IEEE (2009), 43-50.
Other settings are possible, but the different settings of the prior art share common features and limitations.
Regarding the common features, the methods of the prior art use mechanical systems to generate calibrated periodic input to a computer, measure the period of these input and measure the times of output of the periodic input on a display of computer.
Regarding the common limitations:
The invention allows overcoming both drawbacks.
The concept of the invention is to use an interaction loop which is inverted compared to the interaction loop of the methods and systems of the prior art: instead of generating a calibrated input and capturing the corresponding output, according to the invention a calibrated output is generated by a processing device through the output interface, for instance a display; a sensor captures the output at the display and feeds it back to the processing device, thus closing the interaction loop. In the arrangements of the invention, the clock of the processing device synchronizes the interaction loop. A number of arrangements can be used to embody such systems, notably those commented upon further below in the description in relation to
At time 200 an instruction is sent to the processor to move the texture to start a window repaint. At time 201 (not measured in the example of the figure), the texture actually starts moving on-screen. The lag between 200 and 201 is the display lag (drivers and pixels). The change in pixels on-screen is captured by the mouse and an HID report is sent at time 202 (not measured in the example of the figure). An HID report is a message sent from the mouse driver to the processing device which carries the information received by the mouse sensor (for instance, X, Y coordinates of the center of the mouse). The lag between 201 and 202 is the mouse lag (combination of sensor and MCU lag). At time 210, the HID report is received at the processing device. The lag between 202 and 210 is the USB link lag (in case the mouse is connected to a USB port; but of course, it can be connected to other types of ports, for instance a PS2 port). At time 220, the system pointer is moved. The lag between 210 and 220 is the OS lag. At time 230, a MouseMove event is created. At time 240, a MouseMouse event is dispatched. The lag between 220 and 230 and between 230 and 240 is the toolkit lag.
Therefore the end-to-end latency between 200 and 240 can be measured, as well as the latencies of the main components in the interaction loop.
In a first embodiment depicted on
A modern computer mouse consists of a small camera comprising a pixel array comprising typically between 18×18 and 30×30 pixels, a lens and a LED or laser to illuminate the surface. The mouse microcontroller processes the images at a high frame rate, typically between 1500 Hz and 12000 Hz, to detect visible features in the image and measure their translation between consecutive images. The measured displacements are reported to the computer as counts. The sensor has a relative responsiveness above 80% typically for wavelengths between 500 nm (green) and 950 nm (infrared). The camera adjusts its shutter to keep the brightest pixel at a given value. The lens determines the dimensions of the area under the mouse recorded by the camera. Laser mice typically record a smaller area to see finer details.
One of the methods of the invention for measuring end-to-end latency consists in positioning a standard computer mouse at a fixed location on an horizontally oriented monitor. Alternatively, one can use adhesive (e.g. Blu-Tack) to stick the mouse on a vertical monitor but should take care not to increase the distance between the sensor and the screen too much in order to avoid blurring the captured image. Depending on the shininess of the screen surface, the LED of the mouse has to be obscured with e.g. black tape, if the mouse cursor hardly moves or does not move when moving the mouse on the monitor displaying a white background.
Once the mouse is correctly set up on the display, a given texture is displayed on the screen and moved of a controlled distance (e.g. 1 px), while a timestamp is recorded. The texture displacement is intended to create a well-controlled closed loop by producing a fake mouse displacement at a given and measurable time, which will thus be detected and reported by the mouse sensor to the system as a normal event. Upon reception of this event and subsequent ones (drivers, system pointer, toolkit, etc.), successive timestamps are recorded and latency can be measured at several characteristic instants as discussed in relation to
The texture 320a must be selected adequately, so that, when displayed, its translation will produce a displacement that (i) can be detected by the mouse sensor; and (ii) can subsequently produce a system cursor displacement.
Mice resolution is reported in CPI (counts per inch) which corresponds to the smallest displacement they can report. It starts at 400 CPI and can go up to 12,000 CPI. Screens resolution, in PPI (pixels per inch), is far lower, around 100 PPI for current average-end LCD monitors, while high-resolution monitors can go up to 300-400 PPI. This means that a translation of a single pixel on the highest-end monitor is larger than what can measure the lowest-end mouse. As a result, moving a texture of one pixel ensures the displacement can be detected by the mouse. It is also useful to configure the arrangement so that a part of the texture remains visible by the mouse sensor after it is translated: thus the mouse will see common features between two images to compute a displacement.
Accordingly, the displacements of the mouse have to be converted in movements of the texture on the screen using a transfer function which is specific to the OS of the computer. Moving the texture of one pixel is likely to produce mouse reports as low as 1 count. Thus, it is generally useful to configure the arrangement so that 1 count produces a displacement of the system pointer of at least 1 pixel after the operating system transfer function is applied.
In an alternative embodiment depicted on
The impact on latency of the position of the sensor/image arrangement on the screen is a known fixed parameter. In most configurations, the zero position is the upper left corner of the screen.
Sensors of other types can be selected and implemented using the principles of the invention. For example, in the audio domain, one could design a specific ‘audio’ texture/signature, that once played by the processing device through sound monitors could be captured by a microphone connected to the processing device and recognized by the said processing device thanks to recognition algorithms. The processing device would then subtract the time of emission of the audio texture from the time of reception of the audio texture to find the latency.
To determine the elementary pattern 410a of the texture, it is first useful to determine the number of pixels corresponding to an image of the sensor. For example, for an array of 18×18 pixels, the area is about 1 mm2. In a 100 PPI screen, this corresponds to about 3×3 visible pixels. A periodic pattern of this type can therefore advantageously be selected for the texture to be used. On a 300 PPI screen this area corresponds to 9×9 visible pixels. The number N of pixels in an elementary pattern may therefore be 9 or 81 in these examples.
Another element to be taken into effect to select an appropriate texture is the so-called “screen door effect”. To avoid the grid which results from this effect and which can make all measurements impossible, a number of rules have to be applied, which will be discussed further down in the description.
Another element to be taken into account is that it is advantageous to use two types of pixels, 411a and 411b, which have shades with sufficient contrast. For instance, it may be advantageous to set a threshold of contrast above which the differences in shades should be selected. In one embodiment of the invention, it may be advantageous to select pixels 411a of a black shade and pixels 411b of a white shade. In most embodiments, the ratio of contrast between the brighter shade pixels and the darker shade pixels will be higher than 90%, or even 95%.
A combination of these considerations leads to the selection of an advantageous elementary pattern 410a for the texture of the invention, which is displayed on
It can be seen of
The screen door effect results from the fact that the pixels of the display are not exactly contiguous. There is therefore a grid of dark lines which delineates the pixels. Depending on the resolution of the sensor and the distance from the surface of the display, the grid is visible or not. When the sensor is close to the display it is generally visible, like on
The algorithms used to process the signals captured by the sensors of a mouse are not entirely known. It is nevertheless very likely that some characteristic points of high contrast are selected and tracked when the mouse moves (or in our case when the texture moves). The dark lines resulting from the screen door effect will therefore generate characteristic points. Since it is advantageous to have a majority of the characteristic points which are not invariant by a translation of one pixel in the direction of motion, it is necessary to make sure that the characteristic points generated by the screen door effect and are invariant, are as few as possible.
In the measurement arrangement of the invention, a texture of the kind of
By way of example,
Also a comparison of
It can be seen that the number of characteristic points of the third type outweighs by far the number of points of the two other types, which means that this elementary pattern is not fit for the invention.
The pattern of
It can be seen that all the characteristic points, 720b, are correctly tracked. This is why it is specifically advantageous to implement the invention.
For a given texture, the measured values of latency will notably depend on:
The table below displays the success rate (SR) of the measurement, the mean value of end-to-end latency (i.e. the difference in time between the Window repaint event 200 and the MouseMove event 240 of
The graphs of
Latency will also vary significantly from one application to another, as can be seen from the table below.
Also, latency will vary significantly as a function of system load, as evidence by the next table below.
As already explained, a latency of 50 ms or more and a jitter of 20 ms or more will create a bad interaction experience for the user. It becomes all the more important to compensate latency to improve the user experience.
It is possible to take into account these different factors, whether determined by the configuration of the system (hardware and software) or varying over time (like application running and computing load), to model the behavior of a processing device. The model can be represented by a function with parameters or by abaci and can be then used in an operational system to define a calibration model and/or table and then compensate the actual latency and/or jitter upon execution of a defined application in a specific configuration of the processing device. A model fit to implement the invention may use a Kalman filter or Taylor series development algorithm. But other models known to a man of ordinary skill may be used, including statistical models.
It is also possible to use a calibration model or table defined using a specific input interface and a specific output interface and then convert the calibration model or table into another model/table for another pair of input interface/output interface. For some pairs of input interface/output interface, it is possible to define an offset. The offset may be calculated from sets of measurements and statistical adjustments, and then used in a conversion table.
A processing device 910 has an input interface 920 and an output interface 630. In a number of embodiments, the input interface may be a mouse, for instance an optical mouse. The output interface may be a screen. But other arrangements are possible. An input interface may be a touch screen or an audio device. An output interface may be an audio device. Latency may be calibrated using measurements of the same input and output interfaces of the processing device or using measurements from other input and output interfaces than the ones of the processing device and a conversion model or table. Exemplary embodiments of measurements and calibration have been discussed earlier in the description. Calibration is performed either using a model taking into account the various factors which have an impact on latency, for instance the hardware and software configuration of the processing device, as discussed earlier in earlier passages of the description.
In a number of embodiments of the invention, a compensation routine 640 is implemented from the calibration routine 950.
In a number of embodiments, the calibration routine is performed off-line and its results are stored in a table. The table may include values corresponding to a number of parameters of the hardware and/or software configuration of the processing device (input interface, output interface, development environment, application, etc.) and/or variables available on execution of a defined application (e.g. the computing load).
In a number of other embodiments, the calibration routine is executed in real time and its results are passed to the compensation routine. In these embodiments, the arrangement of measurements of latency will be operated upon execution of the application being executed on the processing device.
In a number of embodiments, the measurements performed with the input interface and the output interface which are operative on the processing device may be combined with a conversion model or table resulting from a calibration performed earlier with other input and/or output interfaces.
Examples of hardware and software configurations used for measurements of latency have been disclosed in other parts of the description, which may be used in any combination.
The examples disclosed in this specification are only illustrative of some embodiments of the invention. They do not in any manner limit the scope of said invention which is defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 15306665.9 | Oct 2015 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2016/074848 | 10/17/2016 | WO | 00 |
