Patent Grant
9625999
1. Field
Aspects of embodiments of the present invention relate to an integrated input control and output rendering system.
2. Related Art
The proliferation of processor-driven electronic devices, including, for example, flat screen display devices, touch screens, and “smart” devices (such as smartphones, tablet computers, etc.), has caused a great increase in the replacement of traditional electronic devices (e.g., televisions, cameras) with more versatile flat screen display devices, touch screens, and applications (or “apps”) that run on the smartphones and other smart devices.
Such devices, however, may have their own processors and operating systems—together with their associated processes, priorities, interrupts, and polling—that are replacing more traditional dedicated (and specialized) circuitry for input control and output rendering. Accordingly, adequate response time for acknowledging input actions or commands (such as touching a portion of the display, or moving a device such as a camera lens), or for overlaying other information can be challenging for some types of input since the time and overhead needed to go through the stack of operating system routines to process the input action or data and then acknowledge, respond, or display it to the user through an output display or other rendering device (e.g., speakers) can exceed real time response criteria. Further, alternatives to dealing with real time response criteria, such as more abbreviated processing, may provide insufficient functionality to satisfy users.
Such a delay or abbreviated processing may cause the display device, touch screen, or smart device to lag (or appear to lag) behind the user's input or deliver undesired content, which can lead to a poorer user experience. Even if the display device or smart device may be keeping up with the input, the appearance of lag (such as not timely acknowledging a user input command with the appropriate output action, or not displaying overlay data properly) creates the appearance to the user that the display device or smart device is not keeping up. This can lead to situations such as positive feedback (e.g., where the user tries to re-enter the input command, often multiple times) or unnecessary pausing by the user waiting for the appropriate acknowledgements or poorer displayed images and corresponding overlay information.
Embodiments of the present invention are directed to an integrated input control and output rendering system that circumvents the above and other problems of existing systems. Further embodiments are directed to adding feedback logic in the I/O end of electronic devices that can be configured and specified by the applications that run on the electronic devices.
Still further embodiments of the present invention are directed to using an overlay logic engine, such as in a display device, to directly send user device commands, together with a display rendering engine to interpret the user device commands and process them immediately and independently of the device processor or app interface, thereby allowing the appropriate output rendering (such as updating the display device) to take place concurrently with the normal screen image of the display device. Still further embodiments of the present invention are directed to integrating input sensors (such as a keyboard, mouse, touchpad, camera, biological sensor (e.g., heart rate monitor, thermometer for body temperature), etc.) and output actuators (such as a display panel, speaker, robot, etc.) of a processor-driven user device into an integrated I/O device independently from the user device and its associated processor or applications.
In an embodiment of the present invention, an input/output (I/O) processor for controlling I/O of a user device is provided. The user device includes a device processor, an input device, and an output device. The I/O processor includes an input logic engine configured to receive and process input data from the input device without processing by the device processor, an output rendering engine configured to convert the processed input data into corresponding output data and send to the output device without processing by the device processor, and a control interface configured to transfer the processed input data from the input logic engine to the output rendering engine without processing by the device processor.
The input device may include a touch sensor configured to detect a user's touch. The input data may include touch sensor data corresponding to the user's touch. The output device may include a display panel configured to display an image using image data. The output data may include the image data adjusted to reflect the user's touch.
The input device may include a hovering sensor configured to detect a user's hover. The input data may include hovering sensor data corresponding to the user's hover. The output device may include a sound device configured to play sound using sound data. The output data may include the sound data adjusted to reflect the user's hover.
The input device may include a camera configured to detect a user's gesture. The input data may include gesture sensor data corresponding to the user's gesture. The output device may include a display panel configured to display an image using image data. The output data may include the image data adjusted to reflect the user's gesture.
The input device may include a light sensor configured to detect ambient light. The input data may include light sensor data corresponding to the ambient light. The output device may include a display panel configured to display an image using image data. The output data may include the image data adjusted to reflect the ambient light.
The input device may include a network interface configured to supply streaming data and a graphics content as the input data. The output device may include a display panel configured to display an image using image data. The output data may include the streaming data converted into the image data and overlaid with the graphics content. The output rendering engine may be further configured to adjust a display location of the overlaying graphics content according to a content of the image data.
The input device may include a camera configured to supply video data. The input data may include the video data and overlay data supplied from an input source. The output device may include a display panel configured to display an image using image data. The output data may include the video data converted into the image data and overlaid with the overlay data.
The I/O processor may further include a display buffer configured to receive image data from the device processor. The output device may include a display panel configured to display an image using the image data. The control interface may include an overlay buffer configured to transfer the processed input data as overlay data. The output rendering engine may be further configured to overlay the image data from the display buffer with the overlay data from the overlay buffer. The output data may include the image data as overlaid by the overlay data.
In another embodiment of the present invention, a method of controlling input/output (I/O) of a user device by an I/O processor is provided. The user device includes a device processor, an input device, and an output device. The I/O processor includes an input logic engine, an output rendering engine, and a control interface. The method includes: receiving and processing, by the input logic engine, input data from the input device without processing by the device processor; converting, by the output rendering engine, the processed input data into corresponding output data and sending to the output device without processing by the device processor; and transferring, by the control interface, the processed input data from the input logic engine to the output rendering engine without processing by the device processor.
The input device may include a touch sensor configured to detect a user's touch. The output device may include a display panel configured to display an image using image data. The method may further include: receiving, by the input logic engine, touch sensor data corresponding to the user's touch as the input data from the touch sensor; and adjusting, by the output rendering engine, the image data to reflect the user's touch and sending the adjusted image data as the output data to the display panel.
The input device may include a hovering sensor configured to detect a user's hover. The output device may include a sound device configured to play sound using sound data. The method may further include: receiving, by the input logic engine, hovering sensor data corresponding to the user's hover as the input data from the hovering sensor; and adjusting, by the output rendering engine, the sound data to reflect the user's hover and sending the adjusted sound data as the output data to the sound device.
The input device may include a camera configured to detect a user's gesture. The output device may include a display panel configured to display an image using image data. The method may further include: receiving, by the input logic engine, gesture sensor data corresponding to the user's gesture as the input data from the camera; and adjusting, by the output rendering engine, the image data to reflect the user's gesture and sending the adjusted image data as the output data to the display panel.
The input device may include a light sensor configured to detect ambient light. The output device may include a display panel configured to display an image using image data. The method may further include: receiving, by the input logic engine, light sensor data corresponding to the ambient light as the input data from the light sensor; and adjusting, by the output rendering engine, the image data to reflect the ambient light and sending the adjusted image data as the output data to the display panel.
The input device may include a network interface configured to supply streaming data and a graphics content as the input data. The output device may include a display panel configured to display an image using image data. The method may further include: receiving, by the input logic engine, the streaming data and the graphics content as the input data from the network interface; converting, by the output rendering engine, the streaming data into the image data; overlaying, by the output rendering engine, the image data with the graphics content and sending the overlaid image data as the output data to the display panel; and adjusting, by the output rendering engine, a display location of the overlaying graphics content according to a content of the image data.
The input device may include a camera configured to supply video data. The output device may include a display panel configured to display an image using image data. The method may further include: receiving, by the input logic engine, video data from the camera and overlay data from an input source; converting, by the output rendering engine, the video data into the image data; and overlaying, by the output rendering engine, the image data with the overlay data and sending the overlaid image data as the output data to the display panel.
The I/O processor may further include a display buffer configured to receive image data from the device processor. The output device may include a display panel configured to display an image using the image data. The control interface may include an overlay buffer configured to transfer the processed input data as overlay data. The method may further include overlaying, by the output rendering engine, the image data from the display buffer with the overlay data from the overlay buffer and sending the overlaid image data as the output data to the display panel.
In yet another embodiment of the present invention, a display device is provided. The display device includes a display device processor, an input device, a display panel configured to display an image using image data, and an input/output (I/O) processor for controlling I/O of the display device. The I/O processor may include an input logic engine configured to receive and process input data from the input device without processing by the display device processor, an output rendering engine configured to convert the processed input data into corresponding said image data and send to the display panel without processing by the display device processor, and a control interface configured to transfer the processed input data from the input logic engine to the output rendering engine without processing by the display device processor.
The input device may include a touch sensor configured to detect a user's touch. The input data may include touch sensor data corresponding to the user's touch. The image data may be adjusted to reflect the user's touch.
The input device may include a camera configured to detect a user's gesture. The input data may include gesture sensor data corresponding to the user's gesture. The image data may be adjusted to reflect the user's gesture.
The input device may include a light sensor configured to detect ambient light. The input data may include light sensor data corresponding to the ambient light. The image data may be adjusted to reflect the ambient light.
Embodiments of the present invention avoid the drawbacks of processor-driven user devices, such as display devices, touch screens, and smartphones, whose input control and corresponding output rendering response times may not maintain real time performance (or only maintain it at the expense of acceptable functionality), which can lead to mistakes, positive feedback, and poor user experience.
The accompanying drawings, together with the specification, illustrate example embodiments of the present invention. These drawings, together with the description, serve to better explain aspects and principles of the present invention.
Example embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings, the same or similar reference numerals refer to the same or similar elements throughout. Herein, the use of the term “may,” when describing embodiments of the present invention, refers to “one or more embodiments of the present invention.” In addition, the use of alternative language, such as “or,” when describing embodiments of the present invention, refers to “one or more embodiments of the present invention” for each corresponding item listed.
In one or more embodiments, an integrated input control and output rendering system is provided. The integrated input control and output rendering system may be part of a processor-driven user device, such as a display device, touch screen, or smartphone. The integrated input control and output rendering system may shortcut the user device processor (or processors), offloading input/output (I/O) operations normally performed by the user device processors to improve response time, functionality, or otherwise improve the user experience. For example, the integrated I/O rendering system may include one or more I/O processors dedicated to performing I/O operations. On a smart device, for instance (like a smartphone), these I/O processors may circumvent the usual application-driven approach of the smart device, where a central processor or group of processors treats everything like a user application (or app), with corresponding prioritizing and other operating-system-specific processing.
Referring to
The input device 220 may be any input device capable of receiving input and generating corresponding input signals as the input data. For example, the input device 220 may be a touch sensor configured to sense a user's touch, with the input data being touch sensor data corresponding to the user's touch. In another embodiment, the input device 220 may be a hovering sensor configured to detect a user's hover, with the input data being hovering sensor data corresponding to the user's hover. In yet another embodiment, the input device 220 may be a camera configured to detect a user's gesture, with the input data being gesture sensor data corresponding to the user's gesture.
In another embodiment, the input device 220 may be a light sensor configured to detect ambient light, with the input data being light sensor data corresponding to the ambient light. In yet another embodiment, the input device 220 may be a network interface configured to supply streaming data and a graphics content as the input data. In still yet another embodiment, the input device 220 may be a camera configured to supply video data, with the input data being the video data and overlay data supplied from an input source.
In another embodiment, the input device may be a biological sensor, such as a heart rate monitor or thermometer for measuring body temperature. For instance, the heart rate monitor may monitor a user's heart rate, providing input data including the user's instantaneous or average heart rate, or individual pulses. The thermometer may provide input data including the user's present or average body temperature.
The output device 230 may be any output device capable of receiving output data and outputting corresponding output signals. For example, the output device 230 may be a display panel configured to display an image using image data, with the output data being the image data adjusted to reflect a user's touch. In another embodiment, the output device 230 may be a sound device configured to play sound using sound data, with the output data being the sound data adjusted to reflect a user's hover. In yet another embodiment, the output device 230 may be a display panel configured to display an image using image data, with the output data being the image data adjusted to reflect a user's gesture. In still yet another embodiment, the output device 230 may be a display panel configured to display an image using image data, with the output data being the image data adjusted to reflect the ambient light.
In another embodiment, the output device 230 may be a display panel configured to display an image using image data, with the output data being streaming data converted into the image data and overlaid with a graphics content. Here, the output rendering engine may be further configured to adjust a display location of the overlaying graphics content according to a content of the image data. In yet another embodiment, the output device 230 may be a display panel configured to display an image using image data, with the output data being video data converted into the image data and overlaid with overlay data.
In another embodiment, the I/O processor 100 may further include a display buffer configured to receive image data from the device processor 210, the output device 230 may be a display panel configured to display an image using the image data, the output rendering engine may be further configured to overlay the image data from the display buffer with overlay data from an overlay buffer, with the output data being the image data as overlaid by the overlay data. Here, the control interface 130 may include the overlay buffer configured to transfer the processed input data as the overlay data.
For example,
While in comparable smartphones, this type of feedback takes place within the operating system and processors of the smartphone (and which competes with all of the other apps on the smartphone), in an embodiment of the present invention, an integrated input control and output rendering device (or integrated I/O device or integrated I/O controller) directly processes the user input touch and immediately outputs the corresponding circle without disturbing the normal application processing taking place on the smartphone.
Likewise, in
In
However, according to embodiments of the present invention, an integrated I/O controller can take the logo as an input signal directly and place it in an appropriate portion of the display panel, factoring in considerations such as optimizing for content layout, color balance, and panel longevity. That is, the output rendering engine of the integrated I/O controller is further configured to adjust a display location of the overlaying graphics content according to a content of the image data, with the output data being the streaming data converted into the image data and overlaid with the graphics content. This can take place directly in the display device via the integrated I/O controller rather than in the image processing components used to provide the content of the display panel.
In
According to an example embodiment of the present invention, an integrated input control and output rendering system (integrated I/O controller) includes input components, output components, a centralized logic unit to process and interpret incoming signals, and a rendering unit to generate and combine display data and feedback effects.
The integrated I/O device 400 may be part of a processor-driver user device 405, such as a smartphone, display device, or touch screen. The user device 405, for example, may run apps and an operating system, display routines, or touch sensing routines on one or more user device processors. The integrated I/O device 400 includes its own dedicated processor (or processors) for doing its processing independently of the user device's processors (and any corresponding routines, operating system, or apps).
Referring to
A dedicated input logic engine 450 that takes input signals (such as sensor data from the sensors 410, passing-in data 420 from the streaming data sources, and display data 480 from the user device 405) and outputs standardized control interface 460 to a dedicated output rendering engine 470 that composes logic outputs into the desired formats according to receiving devices and actuators 430 and user configuration data (such as from the user device 405) may share the same I/O processor in the integrated I/O device 400. By splitting the feedback logic of sensors 410 and other input controls into a separate dedicated processor outside of the main processor (that is dedicated to user applications), faster response times (and faster apparent response times) are achieved, thus improving the user experience.
In an example input logic engine and command interface according to an embodiment of the present invention, there are two basic types of input data: single event data and continuous event data. Single event data refers to input data provided for a single event or time, for example, with a single time-stamp from an activity conduct that takes place once per request through a set or predefined procedure. Examples of single event data may include the input from taking a photograph, measuring a temperature (such as ambient temperature or body temperature), acquiring a location, etc.
Continuous event data, on the other hand, refers to input data provided for multiple events or over a period of time, for example, consecutive data triggered by separate requests to start and end. Examples of continuous event data may include, for example, shooting a video, moving a mouse, following a person, monitoring the temperature, monitoring the user's heart rate, etc.
The input sources may be any data sources including, for example, sensing signals from 1D sensors or 2D cameras (such as from sensors 410), passing-through data from networks and streaming services (such as streaming data 420), and display image data 480 from application devices and graphics hardware (such as from user device 405). The same input source may output data of either event type (e.g., single or continuous) depending on its configuration settings (such as a camera, which may take a still picture or shoot a video).
As shown in
Tracking parameters may refer to combinations of correlated input signals, such as input signals having a same shared association or identity. For example, with touch events, the touches of the same finger may be considered as signals of the same track, or be denoted by the same tracking identification (tracking ID). In a similar manner, the gestures of the same hand may be denoted by the same tracking ID. Here, the tracking parameters may represent actual coordinates.
Thus, both detection messages and tracking parameters may be, for example, direct derivatives of some single input sensor, or more sophisticated combinations of multiple input sensors, or even more intelligent interpretations derived with help from one or more sources of collected data, such as a database or set of individual statistics. According to an example integrated I/O device, a generic/standard interface is provided to handle the general input data.
The output rendering engine 470 takes output 460 of the input logic engine 450 as input and renders the output into the desired format of the individual output components 430. The rendered output may include, for example, bypass commands, device reconfigurations, audio-visual displays, other actuators, or network devices. Bypass commands refers to passing command signals from the input logic engine 450 to the user device 405. Device reconfigurations refers to taking action from those command signals that are related to reconfiguring and resetting various I/O device parameters. Audio-visual displays refers to sending visual data (intended for the main display panel) and sound track data (intended for speakers) in their native resolution and quality, by composing the input data, for example, from user devices, network services, and pre-configured I/O responses, into corresponding visual or audio signals and augmenting (or possibly replacing) existing visual or audio data intended for these same output devices 430.
In addition, other actuators refers to passing or otherwise distributing relevant commands to other connected actuators and robotics. Network devices refers to delivering data and signals (for instance, detection alerts and archive data), to various networks or cloud repositories, such as for system archives or alarm alerts. According to an example embodiment of the present invention, the output data is reformatted to meet the specific formats of the corresponding targeted output devices or components.
The integrated I/O device 500 of
As shown in
For fast (and perhaps in some sense, optimal) visual data rendering, all touch and gesture responses and feedbacks may be first rendered into a dedicated overlay buffer (for example, with transparency alpha channel; that is, the degree of transparency of the overlay may be controllable by a parameter alpha, such as between 0 and 100%). The latest overlay buffer may then be combined with the active display data (for example, the usual video feed intended for the display panel) right before its presentation to significantly reduce or minimize any latency delay (e.g., from the input touches and gestures to the output responses and feedbacks).
For example, as an intermediate embodiment to more fully developed integrated I/O devices discussed above, display centric devices may provide much of the same functionality. As shown in
Thus, display buffer 635, which receives graphical output from the user device 605 (such as from a GPU of the user device 605) has its content overlaid or otherwise merged with that of the overlay buffer 615 by display rendering engine 620, which is configured to overlay the image data from the display buffer 635 with the overlay data from the overlay buffer 615, and then supply the image data (as overlaid by the overlay data) as output data to a display panel 640.
Another embodiment of the present invention is to put the integrated I/O design as a component unit within a user device. That is, a dedicated I/O processor is added to provide the functionality of the input logic engine and the output rendering engine as a separate processor from the main user device processor. This provides for a low latency design where user interaction feedback is handled by the dedicated I/O processor directly without going through the user device processor (such as a main application processor or other processor), together with corresponding display or touch routines, operating system, or user apps competing for processing time on the application processor. Thus, the display (or other rendering) of feedback information is not an aftermath decision from the applications being run on the host CPU of the user device, but rather an instant reaction of the dedicated I/O processor that is based on (or customized to) the device's or applications' configuration.
Processing begins, and in step 710, the input logic engine 110 receives and processes input data from the input device 220 without processing by the device processor 210. That is, the input logic engine 110 does the receiving and processing of the input data from the input device 220 in place of the device processor 210. The device processor 210, however, is still free to do other processing while the input logic engine 110 does the receiving and processing of the input data from the input device 220.
In step 720, the output rendering engine 120 converts the processed input data into corresponding output data and sends to the output device 230 without processing by the device processor 210. That is, the output rendering engine 120 does the converting and sending of the processed input data to the output device 230 in place of the device processor 210. The device processor 210, however, is still free to do other processing while the output rendering engine 120 does the converting and sending of the processed input data to the output device 230.
In step 730, the control interface 130 transfers the processed input data from the input logic engine 110 to the output rendering engine 120 without processing by the device processor 210. That is, the control interface 130 does the transferring of the processed input data from the input logic engine 110 to the output rendering engine 120 in place of the device processor 210. The device processor 210, however, is still free to do other processing while the control interface 130 does the transferring of the processed input, data from the input logic engine 110 to the output rendering engine 120.
In summary, embodiments of the present invention provide for an independent I/O device capable of delivering much lower latency turnaround for processing user input or transactions and their corresponding display (or other output) feedback, allowing, for example, smart content placement to be decided by the display panel (which, for example, may extend display panel longevity), enabling panel specific compensated display (to provide, for example, a power-saving option, local environment color adjustment, etc.), supporting direct AR (aspect ratio) information display from a content data source (such as a predefined content data source) with shortcuts for power saving, supporting low latency and low power data path, and supporting a dual frame feature to balance the requirements of high performance and low power.
While the present invention has been described in connection with certain example embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, and equivalents thereof.
This application claims priority to and the benefit of U.S. Provisional Application 61/899,729, entitled “INTEGRATED INPUT CONTROL AND OUTPUT DISPLAY SYSTEM,” filed on Nov. 4, 2013, the entire content of which is incorporated by reference herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7010634 | Silvester | Mar 2006 | B2 |
| 20030007321 | Dayley | Jan 2003 | A1 |
| 20040217937 | Moskalik | Nov 2004 | A1 |
| 20060146491 | Chen | Jul 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 2007-168649 | Jul 2007 | JP |
| 2012-226670 | Nov 2012 | JP |
| 10-2012-0022239 | Mar 2012 | KR |
| 10-2013-0038544 | Apr 2013 | KR |
| Number | Date | Country | |
|---|---|---|---|
| 20150123945 A1 | May 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61899729 | Nov 2013 | US |
