Patent Application
20140313230
Many computing devices include a display for displaying text and images. For example, a display can be used to display graphical user interfaces, documents, photographs, images, and the like. Manufacturers of these devices are challenged with providing displays that accurately depict text and images to provide a satisfying user experience.
The following detailed description refers to the drawings, wherein:
a) is a depiction of image data as it is intended to appear from a first position relative to a display, according to an example.
b) is a depiction of the image data as it might appear from a second position relative to the display, according to an example.
c) is a depiction of the image data after transformation so as to reduce distortion when viewed from the second position, according to an example.
Manufacturers of computing devices, such as smart phones, tablet and slate computers, laptop computers, and the like, are challenged with enabling consumer friendly usage of these devices. For example, manufacturers are challenged with providing displays that accurately depict text and images to provide a satisfying user experience. As described in detail below, various example embodiments relate to image transformation techniques that can enable the accurate depiction of image data on displays.
For example, during a computing session, a user may cause the display of his computing device to change position or may change his position relative to the display. For instance, a display may be tilted back or forward relative to a line of sight of the user, the user may change his position to the left or right of a centerline of the display, and/or the user may alter a distance between him and the display. Any of these things may be considered to change the user's position relative to the display. As a result of the change of position, image data displayed on the display may appear distorted. For instance, if a display is tilted back relative to a user's gaze, images displayed on the display may appear distorted (e.g., a square may appear as a trapezoid and a circle may appear as a lopsided ellipse).
According to an example embodiment, however, the user's position may be detected and the image data may be transformed. The image data may be transformed so that the transformed image data appears to the user from his position largely as the image data was intended to appear (e.g., as the untransformed image data likely would appear from a default position directly in front of the display). Thus, the transformation may reduce the appearance of distortion when the transformed image data is viewed from the user's position. For instance, in the example above the square may appear as a square rather than a trapezoid and the circle may appear as a circle rather than a lopsided ellipse. Accordingly, image data may be depicted in an accurate manner despite a change in user position, thus providing a satisfying user experience. Further details of this embodiment and associated advantages, as well as of other embodiments and applications, will be discussed in more detail below with reference to the drawings.
Referring now to the drawings,
Computing device 100 may include a display. Display 110 may be any of various display screens. For example, the display may be a display integrated into computing device 100, such as in the case of tablet computers and all-in-one computers. The display may also be a display remotely connected to computing device 100, such as a LCD monitor. Furthermore, the display may be a touch sensitive display. Display 110 may be used for various purposes, such as to display a user interface, text, images, and movies.
Computing device 100 may include memory 120, such as a machine-readable storage medium. The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that contains or stores data and/or executable instructions. Thus, the machine-readable storage medium may comprise, for example, various Random Access Memory (RAM), Read Only Memory (ROM), flash memory, and combinations thereof. For example, the machine-readable medium may include a Non-Volatile Random Access Memory (NVRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage drive, a NAND flash memory, and the like. Further, the machine-readable storage medium can be computer-readable and non-transitory.
Memory 120 may store various data. For example, memory 120 may store image data 122. Image data 122 may be data that is to be displayed on display 110. Image data 122 may include user interfaces, images, text, documents, photographs, videos, and the like.
Image data 122 may be configured such that it appears without distortion from a first position relative to display 110. The term “without distortion” may mean “without noticeable distortion.” Accordingly, “without distortion” may signify that a user may not detect or perceive any visual distortion of the displayed image data if the user is viewing the image data from the first position.
The first position may correspond to a default user position. For example, the default user position may be a position in which the user's eyes are located directly in front of and in the center of display 110. Additionally, the default user position may be one in which the angle of incidence of the user's gaze on the display surface of display 110 is zero degrees. Thus, when image data 122 is displayed on display 110, it may appear without noticeable distortion when viewed from the first position.
Computing device 100 may include a sensor. Sensor 130 may be used to detect and/or determine a second position of a user relative to display 110. The second position may represent a current position or perspective of the user. Sensor 130 may include any of various sensors to determine a position or perspective of the user.
For example, sensor 130 may be a digital camera integrated into computing device 100, attached thereto, or integrated into or attached to a display associated with computing device 100, such as display 110. The camera may be or include an image sensor. For example, the image sensor may be a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The camera may include a processor, such as a digital signal processor, but may also be controlled by a separate processor in computing device 100, such as controller 140. In addition, the camera may be a stereo camera with multiple lenses and image sensors for capturing three-dimensional images. Sensor 130 may also be or include a tilt sensor. The tilt sensor may be embedded in or attached to display 110 and/or computing device 100 and may be configured to detect a degree of tilt of display 110. Other sensors for detecting the position of the display or the user may be used as well, such as a sound detector (e.g., a microphone), a heat sensor (e.g., a thermographic camera), a vibration sensor (e.g., a gravimeter), an orientation sensor (e.g., an accelerometer), an infrared light sensor, and the like. Furthermore, a combination of sensors may be used.
Sensor 130 may include logic, such as machine-readable instructions, for detecting and recognizing objects in an image represented by image data, and determining the position of the objects. Thus, for example, facial recognition techniques may be used to detect a user's face in an image sensed by a digital camera. Furthermore, the location of the user's eyes and pupils may be determined using the facial recognition technology and/or eye detection technology, thus enabling a more accurate determination of a user's perspective. For instance, the user's line of sight may be determined in this manner. Another technique of determining a location of the user's eyes and pupils is to shine infrared light and sense a reflection of the light from the user's eyes. Alternatively, the location of the user's eyes may be estimated based on the location of the user's head. For instance, if a heat sensor is used, the location of the head of the user may be inferred from the heat image and the location of the user's eyes may be estimated based on statistical data regarding the average location of a human being's eyes on his head.
Sensor 130 and any associated logic may be implemented using controller 140, using a digital signal processor of the camera, or using another processor, in combination with instructions stored in memory.
The second position determined or detected by sensor 130 may include one or more components corresponding to different aspects of a position of the user. For instance, the second position may include one or more values, vectors, or the like, each one representing an aspect of the position.
The second position may include as a component a degree of tilt of display 110 relative to a line of sight of the user. For example, a line of sight of the user may be determined using a camera, such as a digital or thermographic camera, and/or a tilt sensor. Other sensors, such as an accelerometer, may be used as well.
The second position may include as a component a location of the user relative to a centerline of display 110. For example, the location of the user relative to a centerline of the display can be determined using a camera, such as a digital or thermographic camera. The location may represent a user's position to the right or left of a vertical centerline of the display. Additionally, the location may represent a user's position above or below a horizontal centerline of the display. These centerlines may be virtual centerlines represented in a grid which is created based on the sensor's field of view, for example. The grid may be represented as a bitmap. The formation of the grid may take into account a relationship of the display to the location of the camera, so that the center of the grid corresponds to a center of the display. The location of the user on the grid may be determined and compared to the centerlines of the grid. The location may be used to determine an angle of incidence of the user's gaze on the display surface. Other sensors, such as an infrared light sensor, may be used as well.
The second position may include as a component a distance of the user from display 110. For example, the distance of the user from the display can be determined using a camera, such as a digital or thermographic camera. The distance can be estimated based on the size of the user in the grid, based on the size of the user relative to the user's surroundings, etc. Other sensors, such as a noise sensor, may be used as well.
The second position may include any one or all of these components. The user's position or perspective may be more accurately determined using all of the components. Other methods of determining a user's position or perspective may be used as well.
Computing device 100 may include controller 140 having image data transformation module 142. Controller 140 may include a processor and a memory for implementing image data transformation module 142. The memory may be any of various machine-readable storage mediums, such as described with respect to memory 120. The processor may include at least one central processing unit (CPU), at least one semiconductor-based microprocessor, at least one digital signal processor (DSP) such as a digital image processing unit, other hardware devices or processing elements suitable to retrieve and execute instructions stored in memory, or combinations thereof. The processor can include single or multiple cores on a chip, multiple cores across multiple chips, multiple cores across multiple devices, or combinations thereof. The processor may fetch, decode, and execute instructions from memory to perform various functions, such as generating, processing, and transmitting image data. As an alternative or in addition to retrieving and executing instructions, the processor may include at least one integrated circuit (IC), other control logic, other electronic circuits, or combinations thereof that include a number of electronic components for performing various tasks or functions.
Image data transformation module 142 may transform image data 122 based on the second position of the user determined by sensor 130. The transformed image data may appear from the second position as image data 122 would appear from the first position. Thus, for example, the transformed image data may appear with less distortion than would have been visible had the image data not been transformed.
However, some distortion or visual defects may not be correctable by image data transformation module 142. For example, if the display is tilted back relative to a user's line of sight, the display area (and, consequently, the image data) may appear with less height than if the display were viewed straight-on. Accordingly, the transformed image data appearing from the second position as the image data would appear from the first position is not meant to signify that the transformed image data appears (or is perceived by the user) exactly as the untransformed image data would appear (or be perceived) from the first position. Rather, it is meant to signify that the transformation reduces some distortion that results from the user's position or perspective relative to the display. Other distortions or defects beyond the one described above may exist that also may not be correctable by image data transformation module 142.
Image data transformation module 142 may transform the image data using a matrix transformation, such as a perspective transformation. For example, position and angle vectors may be determined based on the second position detected by sensor 130. For instance, the distance and angle between the detected position and the center of sensor 130's field of view or of display 110's display area may be determined. The center of sensor 130's field of view or of display 110's display area may be considered the first position. Accordingly, the change in position or perspective from the first position to the second position may be determined. A perspective transform may then be applied to the image data that is to be displayed on display 110 to accommodate for the change in position or perspective. The perspective transform may essentially correct, counteract, or reverse some of the distortion that would otherwise result from viewing the image data from the second position. Display 110 may then display the transformed image data.
Image transformations may be described as sets of mathematical transformation functions represented by mathematical equations. Each pixel in the image data may be transformed based on the mathematical transformation functions to obtain a set of output pixels, which together may represent the transformed image data. For example, each pixel may be associated with a pixel location represented by coordinates (xin, yin). An appropriate transformation function based on the type of distortion to be corrected (e.g., distortion due to a tilt of the display downwards, distortion due to a tilt of the display upwards, distortion due to a location of the user to the left of the vertical centerline, distortion due to a location of the user to the right of the centerline) may be selected. For example, if the display is tilted down, the appropriate transformation function may operate to foreshorten the image data, while if the display is tilted up/back, the appropriate transformation function may do the opposite. A perspective transformation may have one or more constants which determine the amount and smoothness of the perspective transformation effect. These constants may be determined based on the second position. For instance, the difference between the second position and the first position may be used to determine an appropriate value for the one or more constants in order to reduce the distortion. The output of the transformation may be a set of coordinates (xout, yout) for each pixel, which may represent the new location of each pixel. The image data may then be displayed in accordance with the new pixel locations.
In an example in which the second position includes a single component, such as a tilt of display 110, a single transformation may be applied. For example, a perspective transformation based on the tilt of the display may be calculated and applied to the image data. This transformed image data may then be displayed on display 110.
In an example in which the second position includes multiple components, such as a tilt of display 110 and a location of the user to the right or left of a vertical centerline, multiple transformations may be successively applied. For example, a perspective transformation based on the tilt of the display may be first calculated and applied to the image data. Then, a perspective transformation based on the location of the user to the right or left of the vertical centerline may be calculated and applied to the transformed image data. This twice-transformed image data may then be displayed on displayed 110.
In some embodiments, the change in position or perspective (i.e., the difference between the first position and the second position) determined by controller 140 may be compared to a threshold value before applying a transformation. If the amount of change in position or perspective does not meet the threshold value, the transformation may not be applied. In one example, the different components of the position/perspective (e.g., angle of tilt of the display, location of the user relative to a centerline of the display, distance of user from the display) may be individually compared to a corresponding threshold to determine whether to apply a transformation. Accordingly, minor changes in position where distortion of the image data may be small or not detectable by the human eye may be ignored. This can save processing time.
In some examples, if the distance of the user from the display exceeds a threshold value, a perspective transform may not be applied. This is because the appearance of distortion of displayed image data may lessen as the distance of the user from the display increases. In addition, a perspective transform may not be applied if multiple users are detected by sensor 130. This is because the position or perspective of the multiple users may be different, and thus it may be difficult to accommodate the positions/perspectives of both users.
Computing device 100 may include additional components and features as well, beyond those depicted in
a)-3(c) illustrate a sample use case and image data transformation.
In
Display 224 is tilted back relative to a line of sight 230 of user 210. The line of sight 230 may be detected using camera 226 by sensing an image of user 210 and the surroundings and using facial recognition techniques to determine a location of the user's eyes and/or pupils. A line of sight from the user's eyes or pupils to a center of the display 224 or to a center of a field of view of camera 226 may be determined. The tilt of the display screen relative to the line of sight may be determined using an input from the tilt sensor in conjunction with the determined line of sight of the user. Alternatively, the tilt of the display screen relative to the line of sight may be determined using only the image data from camera 226 by using facial and object recognition techniques.
Camera 226 may also detect a location of user 210 relative to display 224, such as relative to a centerline 240 and/or a centerline 250 of display 224. Additionally, camera 226 may detect a distance 260 of user 210 from display 224.
In this example, the location of the user relative to centerlines 240, 250 and the distance 260 of the user from display 224 are within a threshold amount of a first position associated with image data to be displayed on display 224. However, the tilt of the display relative to line of sight 230 of user 210 is determined to exceed a threshold value. Accordingly, a perspective transform may be used to reduce distortion that may otherwise result due to viewing the image data from the user's detected position.
a) depicts image data 310 including a circle 320 and a square 330 as the image data may appear to user 210 if user 210 is in a first position. The first position may be a default position in which displayed image data does not appear with noticeable distortion. For instance, the first position may be a position in which display 224 is not tilted relative to line of sight 230, user 210 is located essentially in the center of centerlines 240, 250, and user 210 is located at a distance 260 that enables proper viewing of the displayed image data.
b) depicts image data 310 as it may appear to user 210 if user 210 is in a second position. The second position may correspond to the user's position as depicted in
c) depicts transformed image data 350, which is image data 310 after a perspective transform has been applied. To illustrate what happens to the image data as a result of the transformation, transformed image data 350 is shown in
However, when transformed image data 350 is viewed by user 210 on display 224 from the second position, in which the display is tilted back as shown in
Due to the perspective transformation, edges 352, 354 may occur. Edges 352, 354 may result from the slight convergence toward the bottom of image data 350 as a result of the perspective transformation. Edges 352, 354 may be blacked out when image data 350 is displayed on display 224, similar to the use of mattes in letterboxing, which is the practice of transferring film shot in a widescreen aspect ratio to standard-width video format while preserving the film's original aspect ratio. Alternatively, edges 352, 354 may be filled out with additional image data, such as image data that was not originally displayed due to the size of the screen or image data replicated from neighboring pixels of image data 350.
Method 400 may start at 410 where a position of a user may be detected. For example, the position of the user relative to display 110 may be detected using sensor 130 of computing device 100. At 420, it may be determined whether image data would appear distorted from the user's position. For example, the image data may be image data that is to be displayed on display 110. The image data may be stored in a memory, such as image data 122 in memory 120. It can be determined whether the image data would appear distorted from the user's position using controller 140 by determining the differences between the user's position and another position, such as a default position from which the image data would presumably not appear distorted. For example, the differences may include an amount of tilt of display 110 relative to a line of sight of the user, a location of the user relative to one or more centerlines of display 110 or a center of a field of view of sensor 130, and/or a distance of the user from display 110. If the difference(s) between the detected position and the other position exceeds a threshold value, it may be determined that the image data would appear distorted from the user's position.
At 430, the image data may be transformed if it is determined that the image data would appear distorted from the user's position. For example, a perspective transform may be used to transform the image data. In an example, controller 140 may transform the image data using image data transformation module 142. The image data may be transformed using a perspective transformation based on the user's position. At 440, the transformed image data may be displayed. For example, the transformed image data may be displayed on display 110. As a result, the transformed image data may appear less distorted to the user when viewed from the user's position.
Additional functionality may be implemented based on the detection of the user's position.
Method 500 may start at 510 where the position of the user may be detected. For example, the user's position may be detected using sensor 130 of computing device 100. At 520, a display characteristic may be adjusted based on the user's position. For example, the brightness of display 110 may be adjusted by controller 140. For instance, if the user is positioned to one side of display 110, the brightness may be increased so as to increase the visibility of displayed image data from the user's position. Other display characteristics that may be adjusted include contrast, color quality, and the like.
Method 600 may start at 610 where the position of the user may be detected. For example, the user's position may be detected using sensor 130 of computing device 100. At 620, an automatic shutdown of the display may be disabled based on the user's position. For example, controller 140 may disable an automatic shutdown of display 110. Automatic shutdown of the display may mean that the display is dimmed or turned off (e.g., put to sleep). Automatic shutdown of the display may also include when the display is turned off as a result of a sleeping mode, a hibernation mode, or a shutdown mode of computing device 100. The automatic shutdown may have been previously enabled and set by the user, or it may be a default setting of computing device 100. Disabling the automatic shutdown may mean merely pausing or delaying an automatic shutdown action, resetting a time period at the end of which the automatic shutdown is set to occur, or temporarily or completely disabling the automatic shutdown feature. As a result of method 600, a situation may be avoided where a user is actively using computing device 100 (though not necessarily inputting commands) and an automatic shutdown action disrupts his computing session.
Method 700 may start at 710 where an area of focus of the user may be determined. For example, the area of focus may be determined using sensor 130 of computing device 100. In one example, a digital camera may be used to detect which portion of display 110 the user's pupils are focused on. Controller 140 may determine the area of focus using facial recognition techniques. Alternatively, sensor 130 may include a pupil tracker, such as on a pair of eye glasses or contact lenses. At 720, the area of focus may be displayed in a zoomed-in manner. For example, display 110 may correlate the user's area of focus with certain image data which is displayed in the determined area of focus on display 110. The correlated image data may be enhanced by zooming. This may permit the user to more easily view the image data that the user was focused on.
Various techniques of displaying the correlated image data may be used. For instance, the correlated image data may be enhanced such that the correlated image data occupies the entire display area of display 110. In addition, other image data not visible on the screen may be scrolled into view and similarly enhanced by the user shifting his gaze up or down and/or to the left and right. Alternatively, the correlated image data may be enhanced such that it overlays the image data surrounding it. Other techniques may be used as well. In some examples, the image data may be again displayed normally (i.e., without zooming in) if the user shifts his gaze completely away from the display.
Computer 800 may include a processor 810. Processor 810 may be at least one central processing unit (CPU), at least one semiconductor-based microprocessor, at least one digital signal processor (DSP) such as a digital image processing unit, other hardware devices or processing elements suitable to retrieve and execute instructions stored in machine-readable storage medium 820, or combinations thereof. Processor 810 can include single or multiple cores on a chip, multiple cores across multiple chips, multiple cores across multiple devices, or combinations thereof. Processor 810 may fetch, decode, and execute instructions 822, 824, 826 to implement image data transformation. As an alternative or in addition to retrieving and executing instructions, processor 810 may include at least one integrated circuit (IC), other control logic, other electronic circuits, or combinations thereof that include a number of electronic components for performing the functionality of instructions 822, 824, 826. Accordingly, processor 810 may be implemented across multiple processing units and instructions 822, 824, 826 may be implemented by different processing units in different areas of computer 800.
Machine-readable storage medium 820 may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, the machine-readable storage medium may comprise, for example, various Random Access Memory (RAM), Read Only Memory (ROM), flash memory, and combinations thereof. For example, the machine-readable medium may include a Non-Volatile Random Access Memory (NVRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage drive, a NAND flash memory, and the like. Further, the machine-readable storage medium 820 can be computer-readable and non-transitory. Machine-readable storage medium 820 may be encoded with a series of executable instructions for transforming image data.
The instructions 822, 824, 826, when executed by processor 810 (e.g., via one processing element or multiple processing elements of the processor) can cause processor 810 to perform processes, for example, the processes depicted in
Position determination instructions 822 may cause processor 810 to determine a position of a user using sensor 830. Sensor 830 may be or include a digital camera, a thermographic camera, a tilt sensor, or the like. Other sensors may be used as well. The determined position can be relative to display 840. Display 840 may be integrated into computer 800 or attached or remotely connected to computer 800. Display 840 may be a touch screen. The user's position can include multiple components. For example, the position may include a degree of tilt of display 840 relative to a line of sight of the user, a location of the user relative to display 840, and a distance of the user from display 840.
Transformation instructions 824 may cause processor 810 to apply a transformation to image data. The image data can be image data that is to be displayed on display 840. The transformation can be based on the user's position. For instance, the transformation may be a perspective transformation designed to reduce an appearance of distortion of the image data from the user's position. In some examples, the transformation may be applied to the image data only if the user's position differs from another position by a threshold amount. For example, the other position may be a default position. The default position may be a position from which the image data would likely appear without distortion.
Display instructions 826 may cause processor 810 to display the transformed image data on display 840.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US2011/066200 | 12/20/2011 | WO | 00 | 5/13/2014 |
