Patent Application
20230290046
N/A
A scene in three-dimensional (3D) space may be viewed from multiple perspectives depending on the viewing angle. In addition, when viewed by a user with stereoscopic vision, multiple views representing different perspectives of the scene may be perceived contemporaneously, effectively creating a sense of depth that may be perceived by the user. Multiview displays present images having multiple views to represent how scenes are perceived in the 3D world. A multiview display renders different views contemporaneously to provide a realistic experience to the user. Multiview images may be generated and processed dynamically by software. Thereafter they may be rendered by a graphics pipeline in real time. A graphics pipeline may apply a variety of operations on the multiview image when rendering the multiview image for display.
Various features of examples and embodiments in accordance with the principles described herein may be more readily understood with reference to the following detailed description taken in conjunction with the accompanying drawings, where like reference numerals designate like structural elements, and in which:
Certain examples and embodiments have other features that are one of in addition to and in lieu of the features illustrated in the above-referenced figures. These and other features are detailed below with reference to the above-referenced figures.
Examples and embodiments in accordance with the principles described herein provide techniques to improve the users experience of perceiving multiview images by tilting the plane of convergence. As a default, the plane of convergence is typically in parallel with the camera lens at some distance away from the camera. Objects intersected by the plane of convergence appear in focus such that there is no disparity between the different views of such objects. However, when the point of view changes to an extreme angle such as, for example, a bird's-eye view angle, objects of interest in the multiview image may be subject to disparity in a manner that negatively impacts the viewing experience. As described herein, embodiments are directed to tilting the plane of convergence to improve the way objects are perceived based on the angle of the camera's view. In the case of a bird's-eye view, the plane of convergence may be tilted so that it is substantially parallel with the ground. As a result, objects closer to the ground may be perceived with more clarity from the viewer's perspective.
Embodiments are directed to applying graphics-level operations to tilt the plane of convergence in a real-time graphics pipeline. For example, the plane of convergence may be tilted as part of a post-processing operation as the multiview image is rendered in real time. A shader in a graphics pipeline may be configured to apply a shear function to different views to effectively tilt the plane of convergence. For example, the shader may sample pixels from a view of a multiview image in a manner that shears the resulting view. The amount of shearing may correspond to the degree that a particular view is distanced away from a center point of view. To this end, the shear function shears different views of a multiview in real time to effectively tilt the plane of convergence, thereby creating a better viewing experience for the viewer.
In some embodiments, as a viewer physically changes her viewing angle with respect to the multiview display 112, the viewer's eyes may catch different views 106 of the multiview image 103. As a result, the viewer may interact with the multiview display 112 to see different views 106 of the multiview image 103. For example, as the viewer moves to the left, the viewer may see more of the left side of the buildings in the multiview image 103. The multiview image 103 may have multiple views 106 along a horizontal plane and/or have multiple views 106 along the vertical plane. Thus, as a user changes the viewing angle to see different views 106, the viewer may gain additional visual details of the multiview image 103. When processed for display, the multiview image 103 is stored as data in a format that records the different views 106.
As discussed above, each view 106 is presented by the multiview display 112 at different, corresponding view directions 109. When presenting the multiview image 103 for display, the views 106 may actually appear on or in a vicinity of the multiview display 112. A 2D display may be substantially similar to the multiview display 112, except that the 2D display is generally configured to provide a single view (e.g., only one of the views) as opposed to the different views 106 of the multiview image 103.
Herein a ‘two-dimensional display’ or ‘2D display’ is defined as a display configured to provide a view of an image that is substantially the same regardless of a direction from which the image is viewed (i.e., within a predefined viewing angle or range of the 2D display). A conventional liquid crystal display (LCD) found in many smart phones and computer monitors are examples of 2D displays. In contrast herein, a ‘multiview display’ is defined as an electronic display or display system configured to provide different views of a multiview image in or from different view directions contemporaneously from the user's perspective. In particular, the different views 106 may represent different perspective views of a multiview image 103.
The multiview display 112 may be implemented using a variety of technologies that accommodate the presentation of different image views so that they are perceived contemporaneously. One example of a multiview display is one that employs diffraction gratings to control the principle angular directions of the different views 106. According to some embodiments, the multiview display 112 may be a lightfield display, which is one that presents a plurality of light beams of different colors and different directions corresponding to different views. In some examples, the lightfield display is a so-called ‘glasses free’ three-dimensional (3-D) display that may use diffractive gratings to provide autostereoscopic representations of multiview images without the need to special eye wear to perceive depth. In some embodiments, the multiview display 112 may require glasses or other eyewear to control which views 106 are perceived by each eye of the user.
In some embodiments, the multiview display 112 is part of a multiview display system that renders multiview images and 2D images. In this respective, the multiview display system may include a plurality of backlights to operate in different modes. For example, the multiview display system may be configured to provide broad-angle emitted light during a 2D mode using a broad-angle backlight. In addition, the multiview display system may be configured to provide directional emitted light during a multiview mode using a multiview backlight having an array of multibeam elements, the directional emitted light comprising a plurality of directional light beams provided by each multibeam element of the multibeam element array. The multiview display system may be configured to time multiplex the 2D and multiview modes using a mode controller to sequentially activate the broad-angle backlight during a first sequential time interval corresponding to the 2D mode and the multiview backlight during a second sequential time interval corresponding to the multiview mode. Directions of directional light beams of the directional light beam may correspond to different view directions of a multiview image.
For example, in 2D mode, the broad-angle backlight may generate images so that the multiview display system operates like a 2D display. By definition, ‘broad-angle’ emitted light is defined as light having a cone angle that is greater than a cone angle of the view of a multiview image or multiview display. In particular, in some embodiments, the broad-angle emitted light may have a cone angle that is greater than about twenty degrees (e.g., >±20°). In other embodiments, the broad-angle emitted light cone angle may be greater than about thirty degrees (e.g., >±30°), or greater than about forty degrees (e.g., >±40°), or greater than about fifty degrees (e.g., >±50°). For example, the cone angle of the broad-angle emitted light may be greater than about sixty degrees (e.g., >±60°).
The multiview mode may use a multiview backlight instead of a broad-angle backlight. The multiview backlight may have an array of multibeam elements that scatter light as plurality of directional light beams having principal angular directions that differ from one another. For example, if the multiview display 112 operates in a multiview mode to display a multiview image having four views, the multiview backlight may scatter light into four directional light beams, each directional light beam corresponding to a different view. A mode controller may sequentially switch between 2D mode and multiview mode so that a multiview image is displayed in a first sequential time interval using the multiview backlight and a 2D image is displayed in a second sequential time interval using the broad-angle backlight.
In some embodiments, the multiview display system is configured to guide light in a light guide as guided light. Herein, a ‘light guide’ is defined as a structure that guides light within the structure using total internal reflection or ‘TIR’. In particular, the light guide may include a core that is substantially transparent at an operational wavelength of the light guide. In various examples, the term ‘light guide’ generally refers to a dielectric optical waveguide that employs total internal reflection to guide light at an interface between a dielectric material of the light guide and a material or medium that surrounds that light guide. By definition, a condition for total internal reflection is that a refractive index of the light guide is greater than a refractive index of a surrounding medium adjacent to a surface of the light guide material. In some embodiments, the light guide may include a coating in addition to or instead of the aforementioned refractive index difference to further facilitate the total internal reflection. The coating may be a reflective coating, for example. The light guide may be any of several light guides including, but not limited to, one or both of a plate or slab guide and a strip guide. The light guide may be shaped like a plate or slab. The light guide may be edge lit by a light source (e.g., light emitting device).
In some embodiments, the multiview display system is configured to scatter out a portion of the guided light as the directional emitted light using multibeam elements of the multibeam element array, each multibeam element of the multibeam element array comprising one or more of a diffraction grating, a micro-refractive element, and a micro-reflective element. In some embodiments, a diffraction grating of a multibeam element may comprise a plurality of individual sub-gratings. In some embodiments, a micro-reflective element is configured to reflectively couple or scatter out the guided light portion as the plurality of directional light beams. The micro-reflective element may have a reflective coating to control the way guided light is scattered. In some embodiments, the multibeam element comprises a micro-refractive element that is configured to couple or scatter out the guided light portion as the plurality of directional light beams by or using refraction (i.e., refractively scatter out the guided light portion).
As illustrated in
The camera 121 may also represent a virtual (e.g., simulated or hypothetical) camera as opposed to a physical camera. The scene may be generated using computer graphics techniques that manipulate computer-generated information. In this example, the camera 121 is implemented as a virtual camera having a point of view to generate the scene using software tools for editing images. The virtual camera may be defined in terms of viewing angle and coordinates of 3D model. The 3D model may define various objects that are captured by the virtual camera.
In some embodiments, one or more views of the multiview image 115 may be generated through automated algorithms (e.g., computer vision, artificial intelligence, image batch processing, etc.). For example, after generating or capturing views of a scene using a physical or virtual camera, one or more other views may be generated artificially by predicting, interpolating, or extrapolating from the original view. For example, various computer vision techniques may generate additional views based on one or more input views. This may involve employing a trained computer vision model that predicts, interpolates, and/or extrapolates different views from one or more input views.
When generating or capturing views of a scene using the camera 121, the multiview image may have a plane of convergence 127. A ‘plane of convergence’ or ‘convergence plane’ is defined as a set of locations where the different views align such that there is little to no disparity between the different views. The plane of convergence 127 may also be referred to as a Zero Disparity Plane (ZDP). The plane of convergence 127 occurs in front of the camera 121. Objects between the camera 121 and the plane of convergence 127 appear closer to the viewer while objects behind the plane of convergence 127 appear further away from the viewer. In this respect, the degree of disparity between the different views increases the further the object is positioned away from the plane of convergence 127. Objects along the plane of convergence 127 appear in focus with respect to the viewer. Thus, when generating a multiview image 115, the creator who wishes to feature certain objects as the primary subject may want the plane of convergence 127 to fall on the primary subject. Pixels rendered on the ZDP may appear as if they are located on the display, pixels rendered in front of the ZDP may appear as they are located in front of the display, and pixels rendered behind the ZDP may appear as if they are located behind the display.
The camera 121 captures a scene that falls within the frustum 130 of the camera 121. The frustum 130 is shown to have an upper bound and a lower bound that define the viewing angle range of the scene. Typically, the default plane of convergence 127 is parallel to the plane formed by the camera lens of the camera 121 such that it forms a trapezoid with respect to the frustum 130. In
Herein a ‘disparity map’ is defined as information indicating an apparent pixel difference between at least two views of a multiview image 115. In this respect, a disparity map may indicate the difference in location between two pixels of two views of a multiview image. When disparity is zero (e.g., equal to zero or near zero) the pixels representing an object appear to the viewer at the same location. In other words, an object that is in focus by the user has zero disparity between multiple views (e.g., a left eye view and a right eye view). Areas where there is little to no disparity are considered to correspond to the plane of convergence 127 (or ZDP). Objects that appear in front or behind the object in focus will have disparity at varying degrees of disparity and are thus beyond the plane of convergence. For example, pixels representing objects between the camera 121 and the plane of convergence 127 may have positive disparity values while pixels representing objects behind the plane of convergence 127 may have negative disparity values. The larger the absolute value of disparity, the farther it is away from the plane of convergence 127. Disparity is inversely proportional to depth.
By applying a tilted plane of convergence 138, the multiview image may result in a more aesthetically pleasing viewing experience. For example, the tilted plane of convergence 138 may correspond to a plane formed by the ground 120. As a result, objects along the ground will have no disparity, thereby drawing the viewer's attention towards the ground as it spans the multiview image. For example, objects located on or near the ground will appear as if it is on the display, objects above the ground will appear in front of the display and objects below the ground will appear behind the display.
In terms of a mathematical relationship, the plane of convergence 127 may be tilted along the vertical (y) axis by modifying the disparity map according to the following equation (1):
D′(X,Y)=D(X,Y)+T*Y+C (1)
where ‘D’ refers to the disparity value, ‘D′’ refers to the updated disparity value, where the disparity value is a function of the X and Y coordinates of the pixel, where ‘T’ quantifies the amount of tilt 141, and ‘C’ corresponds to the location of the rotation of the plane of convergence 127 defined by a rotation axis 150. By applying the equation above to the disparity map 133, the disparity is modified along the vertical axis such that the further away from the rotation axis 150, the greater the change in disparity.
Modifying disparity maps 133 to create a tilted plane of convergence 138 may not be an option in some embodiments as disparity maps 133 might not be readily available. For example, in a real-time rendering environment, multiview images may be rendered and post-processed on the fly when there is no bandwidth or capability of generating disparity maps. To this end, operating on disparity maps might not allow for real-time rendering in a graphics pipeline. The following Figures describe using a graphics pipeline to tilt the plane of convergence 127 in a real-time rendering environment.
When generating or rendering multiview images, there are various visual properties or effects that control the way the image is displayed. These visual properties include, for example, disparity, depth of field (DoF), baseline, plane of convergence, convergence offset, transparency, etc. The visual properties of a multiview image may be applied at the time of rendering as a post-processing operation.
As used herein, ‘disparity’ is defined as the difference between at least two views of a multiview image at corresponding locations. For example, in the context of stereoscopic vision, the left eye and right eye may see the same object but at slightly different locations due to the difference in viewing angles between the eyes. This difference may be quantified as disparity. The change in disparity across the multiview image conveys a sense of depth.
As used herein, ‘depth of field’ is defined as the difference in depth between two objects that are considered to be in focus. For example, a large depth of field of a multiview image results in a small amount of disparity between a relatively large range of depths.
As used herein, ‘baseline’ or ‘camera baseline’ is defined as the distance between two cameras that capture corresponding views of a multiview image. For example, in the context of stereoscopic vision, the baseline is the distance between the left eye and right eye. A larger baseline leads to increased disparity and may enhance the 3D effect of the multiview image.
As used herein, ‘convergence offset’ refers to the distance between the camera and point along the plane of convergence. Modifying the convergence offset will change the location of the plane of convergence so as to refocus the multiview image on new objects at a different depth.
As used herein, ‘transparency’ refers to an object property that defines the degree in which other objects behind an object may be seen. Objects may be rendered as layers that form a finalized view of the multiview image. Increasing the transparency of a front layer will allow rear layers to be seen. A minimum transparency (e.g., no transparency) will prevent a rear layer from being seen while a maximum transparency will make the particular layer invisible so as to completely reveal the rear layer.
Further, as used herein, the article ‘a’ is intended to have its ordinary meaning in the patent arts, namely ‘one or more’. For example, ‘a processor’ means one or more processor and as such, ‘the memory’ means ‘one or more memory components’ herein.
The computing system shown in
The application 205 may be a user-level application that generates a user interface that is rendered by the graphics pipeline 200 for display on the multiview display 112. For example, the application 205 may be a navigation application that loads various maps depicting streets, buildings, and other geographic landmarks. A navigation application may provide a user interface that generates a 3D model of a geographic area. The navigation application may dynamically update a virtual camera's viewing angle in the 3D model to generate a visual output of a portion of the 3D model based on the virtual camera's orientation.
The computing system may also include memory 208. The memory 208 may include main memory (e.g., system memory), cache, or other fast memory for quickly processing data. The memory 208 may be volatile memory but may also include non-volatile memory, as discussed in further detail below. The memory 208 may include memory for the CPU 202 and memory for the graphics pipeline 200 such that the CPU 202 and graphics pipeline 200 share the same memory resources. In some embodiments, the memory 208 includes a first memory (e.g., CPU memory) that is dedicated to the CPU and includes second memory (e.g., GPU memory, texture memory, etc.) that is dedicated to the graphics pipeline 200. In this embodiment, the graphics pipeline 200 may load, copy, or otherwise move content from CPU memory to GPU memory.
As discussed above the application 205 may generate a 3D model using computer graphics techniques for 3D modeling. A 3D model is a mathematical representation of various surfaces and textures of different objects and may include the spatial relationship between the objects. The application 205 may generate and update the 3D model depending on user input. The user input may involve navigating though the 3D model by clicking or dragging a cursor, pressing direction buttons, converting the user's physical location to a virtual location within the 3D model, etc. The 3D model may be loaded into memory 208 and updated thereafter.
The 3D model may be converted into multiview images 211 revealing a window into the 3D model. The window may be defined by a virtual camera having a set of coordinates within the 3D model, a viewing angle, a focus length, a baseline, etc. A sequence of multiview images 211 may form a video that is displayed at a particular frame rate (e.g., 30 frames per second). Each multiview image 211 may be made up of multiple views 214a-d. The example of
The application 205 may load a view 214a-d of the multiview image 211 into memory 208. For example, the application 205 may be configured to convert the 3D model into a rendered scene for showing a multiview image 211 derived from the 3D model. One or more views 214a-d are generated by the application 205 and placed into a particular block of the memory 208. The view 214a-d may be formatted as a bitmap 217 defined by a pixel coordinate system. For example, the view 214a-d may be expressed as a two-dimensional array of bitmaps along a horizontal (X) axis and a vertical (Y) axis. Each pixel in the bitmap 217 has a corresponding location on the display. For example, the upper left-most pixel of the bitmap 217 controls the output of the upper left-most pixel of the display. In addition, each view 214a-d may have a corresponding view index number 220. The view index number 220 may be an ordered view number of the view within the multiview image 211. For example, in a four-view multiview format, each of the four views may be numbered one, two, three, and four. The view index number 220 indicates the position of the view relative to other views. For example, View One may be the left-most view, View Two may be the left-center view, View Three may be the right-center view, and View Four may be the right most view. The greatest disparity, in this case, would be between View One and View Four.
Once the views 214a-d are generated and loaded into memory 208, the application 205 may invoke a render command 221 to the graphics pipeline 200. The render command 221 instructs the graphics pipeline 200 to begin rendering the multiview image 211. The render command 221 may be a function call to the graphics driver to cause the graphics pipeline 200 to render the multiview image 211. The render command 221 may identify the specific multiview image 211 to be rendered. For example, the render command 221 may identify the address block of where the multiview image 211 is stored.
The graphics pipeline 200 may include one or more shaders 226 to render the multiview image 211. A shader 226 may be a hardware device (e.g., a shader core), a software module, or a combination thereof. A shader 226 may be executed by the GPU of the graphics pipeline 200. An initial rendering of the multiview image 211 may be performed by a module that performs various techniques such as, for example, rasterization, to render a simple or rough version of the multiview image 211. The initial rendering operation may be a quick, highly efficient operation, to convert the scene geometry into pixels for display. The initial rendering may not include more advanced optically advanced effects. In some embodiments, a shader 226 may be used in the initial rendering.
After the initial rendering is performed, one or more advanced optical effects may be applied to the initially rendered multiview image. An optical effect may be applied using one or more shaders 226. By operating on an initially rendered multiview image 211, the shader 226 is considered to implement a post-processing effect. As used herein, ‘post-processing’ is defined as an operation performed on an image that is initially rendered as part of the rendering process in a graphics pipeline 200. Different shaders 226 may be configured to perform post-processing. Some examples of post-processing include, but not limited to, modifying color saturation, modifying hue, adjusting the brightness, adjusting the contrast, applying a blur, performing volumetric lighting, applying a depth effect, performing cell shading, generating a bokeh effect, applying or one or more filters. As used herein, a ‘shader’ is defined is a graphics component in a graphics pipeline that applies a specific graphics operation including, for example, initial rendering or post-processing.
The application 205 may interface with the graphics pipeline 200 using one or more Application Programming Interfaces (API). One example of an API is OpenGL, which provides an interface to allow applications 205 to call functions performed in the graphics pipeline 200. For example, an API may be used by the application 205 to invoke a particular shader 226 that performs post-processing on an initially rendered multiview image 211.
Embodiments are directed to implementing functionality in the graphics pipeline 200 to tilt the plane of convergence during real-time rendering. The following provides an example of the functionality and operations that may take place in the computing system. As discussed above the application 205 may generate and load a view 214a-d of the multiview image 211 into memory 208. The application 205 executing on an operating system may instruct the CPU to load the view 214a-d into a block of memory 208.
The view 214a-d may be formatted as a bitmap 217 defined by a pixel coordinate system. The view 214a-d may be generated from a 3D model by identifying a particular viewpoint and viewing angle of the 3D model and converting it into a bitmap 217. This may be performed for each view of the multiview image 211. The application 205 may then invoke a render command 221 to initially render the views 214a-d of the multiview image 211. For example, the application 205 may use an API to request the graphics pipeline 200 to perform an initial rendering. In response, the graphics pipeline 200 may generate initially rendered views 214a-d by performing, for example, rasterization. In a real-time graphics rendering environment, the graphics pipeline 200 may be optimized to quickly render views 214a-d on the fly. This provides a seamless experience to the viewer as new multiview images 211 are dynamically generated (and are not pre-rendered).
Next, the application 205 is configured to tilt the plane of convergence in real time. For example, the application 205 may identifying a distance between the view 214a-d and a center point of view. Assuming the case where the different views 214a-d have varying degrees of horizontal disparity with respect to a center point of view, the distance between each view and the center point of view along the horizontal axis may be determined. This distance is dependent on the baseline (e.g., the distance between views). For example, the larger the baseline, the greater the distance from a center point of view. In some embodiments, the distance between the view 214a-d and the center point of view is identified by determining an ordered view number (e.g., the view index number 220) of the view 214a-d within the multiview image 211. For example, if the views 214a-d of the multiview image 211 are ordered from one through four, with View One being positioned at the left-most side and View Four being positioned at the right-most side. The view index number 220 corresponds to distance between the view 214a-d and the center point of view. For example, a view index number 220 of one may correspond to a distance of 50 pixels left of the center, a view index number 220 of two may correspond to a distance of 25 pixels left of the center, a view index number 220 of three may correspond to a distance of 25 pixels right of the center, and a view index number 220 of four may correspond to a distance of 50 pixels right of the center. The distance from the center may be a signed number (e.g., positive or negative) to indicate whether the view is left of the center. For example, a negative distance may indicate that the view is left of the center while a positive distance may indicate that the view is to the right of the center.
Determining the view 214a-d distance from the center point of view is part of determining how to tilt the plane of convergence in the real-time graphics pipeline 200. The application 205 may generate a render instruction to tilt the plane of convergence by using a shear function. The application 205 may transmit to the graphics pipeline 200 an instruction to render the view as a sheared view. In this respect, the application 205 may invoke an instruction for post-processing an initially rendered multiview image to be sheared according a shear function applied along an axis of the pixel coordinate system. Specifically, the graphics pipeline 200 may render the view 214a-d in the graphics pipeline 200 as a sheared view according to the shear function. A shear strength of the shear function correlates with the distance between the view 214a-d and the center point of view. A ‘shear function,’ as used herein, is defined as a graphics operation that displaces pixels of an image along a direction according to a shear strength. The shear strength quantifies that amount of shear effect that is applied by a shear function to an image. The shear strength of the shear function may correlate with a position of the view relative to other views in the multiview image.
Executing the shear function causes the plane of convergence to be tilted in real time as the multiview image 211 is rendered in the graphics pipeline. A shader 226 may be customized to implement the shear function. In this respect, the application 205 may call a shader 226 to perform the shear function on an initially rendered multiview image 211. After applying the shear function on the views 214a-d of the multiview image, the graphics pipeline 200 may load the result into memory 208 as a sheared multiview image 232. The graphics pipeline 200 may override the multiview image 211 with the sheared multiview image 232 or may load the sheared multiview image 232 in a separate portion of the memory 208. Additional post-processing may be applied to the sheared multiview image 232 before it is finally rendered on the multiview display 112.
Each view may have a corresponding distance to the center point of view 235. While
The shear function may also be defined by a shear line 238. The shear line 238 may extend along a particular axis which controls how each view is sheared. The shear function operates according to the shear line 238. The example of
The shear line 238 may be form a horizontal line positioned in the middle along the vertical axis as a default. In other embodiments, the shear line 238 may be located at varying vertical locations and may be user-specified. While
One embodiment contemplates using a navigation application to dynamically generate multiview images of map scenes as a user is navigating a physical or virtual space. If the camera angle is similar or close to a bird's-eye view, the plane of convergence may be tilted about a horizontal axis. As a result, the shear function is configured to skew the view only along the horizontal axis of the pixel coordinate system.
The views of the multiview image maybe calculated dynamically in response to user interaction. The application 205 may generate a command to the graphics pipeline 200 to perform real-time rendering of any or all views as they are being calculated by the application 205. For example, the application 205 may transmit an API function call to the graphics pipeline 200 to render the views.
The real-time rendering may include an initial render portion and a post-processing portion. The initial render portion involves the graphics pipeline 200 to render initial views. As discussed above, the views are initially rendered to quickly render the pixel of the multiview image on the display without advanced optical effects. A shader may be used to perform the initial rendering. Thereafter, the application 205 may invoke one or more post-processing operations to convert the initial rendering into a final rendering. The post-processing may apply image-editing operations that improve the quality or realism of the initially rendered image. According to embodiments, the application 205 instructs the graphics pipeline 200 to tilt the plane of convergence. For example, the graphics pipeline 200 applies a shear function to each of the views. A shear strength of the shear function correlates with a position of the view relative to other views in the multiview image. A shader may be used to implement the shear function. The application 205 may provide a shear strength, a shear line, or both as an input to the graphics pipeline. The sheared views of the multiview image are then rendered on the multiview display 112. This process occurs continuously as the application 205 generates new multiview images to be rendered in real time.
The user interface 244 may be rendered on a client device and used by a developer during configuration mode who is developing an application that eventually renders multiview images during run time mode. The user interface may comprise a window containing information (e.g., text and graphics) presented to a user. The user interface 244 may be generated by an application used to design end-user applications. For example, the user interface 244 may be used by developers to design navigation applications, gaming applications, or other applications. The user interface 244 may be used by developers who design graphics and the way they are presented to other users. The user interface 244 may also be rendered by an end-user application. The user interface 244 may allow a user to configure shaders during a configuration mode by making user selections relating to different post-processing operations. Once the shaders are configured according to the user input, the shaders may post-process multiview images in run time.
The user interface 244 may have a first portion 247 for displaying a multiview image or a representation thereof. The first portion 247 may include a rendering of a multiview image. For example, the rendering of the multiview image may simulate how the user settings would apply to multiview images during run time. The user interface 244 may have a second portion 250 that contains a menu. The menu may include various input elements such as, for example, sliders, text boxes, check boxes, radio buttons, dropdown menus, etc.). The menu allows a user to change various visual parameters of the multiview image as it is rendered in the first portion. These visual parameters include, for example, the camera baseline, the convergence offset, a ZDP rotation, an auto ZDP option, a Depth of Field (DoF) Threshold, a DoF strength, a transparency threshold, a transparency strength, and potentially other visual parameters. A user may provide input by manipulating one or more input elements. As a result, user input is received from the user interface 244.
The shear function may also calculate the strength of shear according to the baseline that may be user-specified. The baseline controls the distance between each view and a center point of view by increasing the distance between at least two views. Thus, increasing the baseline may move a view farther away from the center point of view, thereby causing that view to experience a stronger shear effect. To this end, outer views will be sheared to a greater extent to achieve the effect of tilting the plane of convergence.
After selecting the regions of the view, the application applies a shader operation (e.g., a post-processing operation) only to the selected regions. The shader is configured to perform a transparency operation or a depth of field operation, or potentially other post-processing operations. A transparency operation changes the degree in which other objects behind an object may be seen. This degree may be user-specified using the user interface. For example, as shown in
A depth of field operation modifies the difference in depth between two objects that are considered to be in focus. For example, the depth of field operation may change the disparity values for the pixels within the selected pixels. For example, if the depth of field threshold selects pixels having disparity values between −30 and +30, a large depth of field strength may specify the degree of blurriness applied to the selected pixels. The depth of field operation blurs the selected pixels in a manner that corresponds to the depth of field strength.
The user interface 244 allows a user to make particular selections for thresholds and post-processing operation parameters. These settings are used to configure shaders. During run time, the shaders operate according to the settings applied via the user interface 244.
The computing system may load a view of the multiview image into the memory 303. For example, as discussed above with respect to
Next, the computing system may transmit to the graphics pipeline 315 an instruction to render the view (render instruction 317) as a sheared view 320 according a shear function applied along an axis 323 of the pixel coordinate system. The graphics pipeline 315 may be similar to the graphics pipeline 200 of
A shear strength of the shear function correlates with a position of the view 312 relative to other views in the multiview image 309. For example, a view index number may identify the position of the view 312 relative to other views. As explained above, in some embodiments, the shear strength may be determined by a user during a configuration mode who provides user input via a user interface. A shear strength is determined from the user input and is applied during run time.
The graphics pipeline 315 is configured to implement the shear function as pixels of the bitmap are sampled by the graphics pipeline 315. For example, the shear function may involve forming the sheared view by sampling pixels from the view 312. Rather than uniformly sampling in a one-to-one correspondence, the shear function samples pixels along a shear line using a shear strength to cause a shear effect. For example, the shear function operates according to an axis 323 that forms a shear line. Pixels of the sheared view 320 are sampled from locations that are close to the corresponding locations of the view 312. As pixels of the sheared view 320 move farther away from the axis 323 (in the vertical direction), the amount of horizontal displacement increases with respect to where the pixel is being sampled.
To illustrate, pixel D3 of the sheared view 320 is close to the axis, which is near rows 3 and 4. This pixel of the sheared view 320 is sampled from pixel D3 of the view 312. This amounts to no shear effect as pixel sampling is performed at the same corresponding location. However, as pixels are located further upwards in the vertical direction, the shear effect becomes more apparent. Pixel D1 of the sheared view is sampled from pixel Cl of the view 312. In this respect, the pixels north of the axis 323 are skewed to the right. This is a sampling offset that causes the shear effect to be applied to the sheared view. Similarly, pixel D7 of the sheared view 320 is sampled from pixel E7 of the view 312. The pixels south of the axis 323 are skewed to the left. This skewing function may lead to sampling pixels at invalid locations when operating near specific edges of the view 312. For example, pixel G7 of the sheared view 320 is sampled from a location outside of the view (shown as an X). In this case, a default pixel may be used to generate the pixel for G7 of the sheared view 320. The default pixel may be a pixel with a zero color value (e.g., a black pixel) or may have any other default pixel value. In some embodiments, the default pixel value may be determined by matching to the closest pixel that is in bounds.
At item 404, the computing device generates a plurality of views of a multiview image. For example, an application (e.g., application 205 of
At item 407, the computing device identifies a distance between each view and a center point of view. For example, an application may identify this distance based on the view index number that indicates the position of each view relative to another view. The view index number may indicate whether the view is to the right or left of the center and how close it is to the center as index numbers are ordered. The distance may also be calculated according to the baseline. In the event that the baseline is predetermined, the view index number may be sufficient to infer the distance between a view and a center point of view.
At item 410, the computing device applies a shear function to each view to generate sheared views. For example, a graphics pipeline (e.g., graphics pipeline 200 of
At item 413, the computing device displays the rendered sheared views. The sheared views effectively have a tilted plane of convergence controlled by the amount of shearing applied to each view. The views may be rendered as a multiview image on a multiview display. For example, a graphics pipeline may communicate with the multiview display using, for example, graphics drivers and/or firmware to cause the multiview image to be rendered for display.
The flowchart of
Although the flowchart of
A processor 1003 may be a central processing unit (CPU), graphics processing unit (GPU), any other integrated circuit that performs computing processing operations, or any combination thereof. The processor(s) 1003 may include one or more processing cores. The processor(s) 1003 comprises circuitry that executes instructions. Instructions include, for example, computer code, programs, logic, or other machine-readable instructions that are received and executed by the processor(s) 1003 to carry out computing functionality that are embodied in the instructions. The processor(s) 1003 may execute instructions to operate on data. For example, the processor(s) 1003 may receive input data (e.g., an image), process the input data according to an instruction set, and generate output data (e.g., a processed image). As another example, the processor(s) 1003 may receive instructions and generate new instructions for subsequent execution. The processor 1003 may comprise the hardware to implement a graphics pipeline (e.g., graphics pipeline 200 of
The memory 1006 may include one or more memory components. The memory 1006 is defined herein as including either or both of volatile and nonvolatile memory. Volatile memory components are those that do not retain information upon loss of power. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. System memory (e.g., main memory, cache, etc.) may be implemented using volatile memory. System memory refers to fast memory that may temporarily store data or instructions for quick read and write access to assist the processor(s) 1003. The memory 1006 may include the memory 208 of
Nonvolatile memory components are those that retain information upon a loss of power. Nonvolatile memory includes read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via a memory card reader, floppy disks accessed via an associated floppy disk drive, optical discs accessed via an optical disc drive, magnetic tapes accessed via an appropriate tape drive. The ROM may comprise, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other like memory device. Storage memory may be implemented using nonvolatile memory to provide long term retention of data and instructions.
The memory 1006 may refer to the combination of volatile and nonvolatile memory used to store instructions as well as data. For example, data and instructions may be stored in nonvolatile memory and loaded into volatile memory for processing by the processor(s) 1003. The execution of instructions may include, for example, a compiled program that is translated into machine code in a format that can be loaded from nonvolatile memory into volatile memory and then run by the processor 1003, source code that is converted in suitable format such as object code that is capable of being loaded into volatile memory for execution by the processor 1003, or source code that is interpreted by another executable program to generate instructions in volatile memory and executed by the processor 1003, etc. Instructions may be stored or loaded in any portion or component of the memory 1006 including, for example, RAM, ROM, system memory, storage, or any combination thereof.
While the memory 1006 is shown as being separate from other components of the multiview display system 1000, it should be appreciated that the memory 1006 may be embedded or otherwise integrated, at least partially, into one or more components. For example, the processor(s) 1003 may include onboard memory registers or cache to perform processing operations.
I/O component(s) 1009 include, for example, touch screens, speakers, microphones, buttons, switches, dials, camera, sensors, accelerometers, or other components that receive user input or generate output directed to the user. I/O component(s) 1009 may receive user input and convert it into data for storage in the memory 1006 or for processing by the processor(s) 1003. I/O component(s) 1009 may receive data outputted by the memory 1006 or processor(s) 1003 and convert them into a format that is perceived by the user (e.g., sound, tactile responses, visual information, etc.).
A specific type of I/O component 1009 is a display 1012. The display 1012 may include a multiview display (e.g., multiview display 112), a multiview display combined with a 2D display, or any other display that presents images. A capacitive touch screen layer serving as an I/O component 1009 may be layered within the display to allow a user to provide input while contemporaneously perceiving visual output. The processor(s) 1003 may generate data that is formatted as an image for presentation on the display 1012. The processor(s) 1003 may execute instructions to render the image on the display for being perceived by the user.
The bus 1015 facilitates communication of instructions and data between the processor(s) 1003, the memory 1006, the I/O component(s) 1009, the display 1012, and any other components of the multiview display system 1000. The bus 1015 may include address translators, address decoders, fabric, conductive traces, conductive wires, ports, plugs, sockets, and other connectors to allow for the communication of data and instructions.
The instructions within the memory 1006 may be embodied in various forms in a manner that implements at least a portion of the software stack. For example, the instructions may be embodied as an operating system 1031, an application(s) 1034, a device driver (e.g., a display driver 1037), firmware (e.g., display firmware 1040), or other software components. The operating system 1031 is a software platform that supports the basic functions of the multiview display system 1000, such as scheduling tasks, controlling I/O components 1009, providing access to hardware resources, managing power, and supporting applications 1034.
An application(s) 1034 executes on the operating system 1031 and may gain access to hardware resources of the multiview display system 1000 via the operating system 1031. In this respect, the execution of the application(s) 1034 is controlled, at least in part, by the operating system 1031. The application(s) 1034 may be a user-level software program that provides high-level functions, services, and other functionality to the user. In some embodiments, an application 1034 may be a dedicated ‘app’ downloadable or otherwise accessible to the user on the multiview display system 1000. The user may launch the application(s) 1034 via a user interface provided by the operating system 1031. The application(s) 1034 may be developed by developers and defined in various source code formats. The applications 1034 may be developed using a number of programming or scripting languages such as, for example, C, C++, C #, Objective C, Java®, Swift, JavaScript®, Perl, PHP, Visual Basic®, Python®, Ruby, Go, or other programming languages. The application(s) 1034 may be compiled by a compiler into object code or interpreted by an interpreter for execution by the processor(s) 1003. The application 1034 may be the application 205 of
Device drivers such as, for example, the display driver 1037, include instructions that allow the operating system 1031 to communicate with various I/O components 1009. Each I/O component 1009 may have its own device driver. Device drivers may be installed such that they are stored in storage and loaded into system memory. For example, upon installation, a display driver 1037 translates a high-level display instruction received from the operating system 1031 into lower level instructions implemented by the display 1012 to display an image.
Firmware, such as, for example, display firmware 1040, may include machine code or assembly code that allows an I/O component 1009 or display 1012 to perform low-level operations. Firmware may convert electrical signals of particular component into higher level instructions or data. For example, display firmware 1040 may control how a display 1012 activates individual pixels at a low level by adjusting voltage or current signals. Firmware may be stored in nonvolatile memory and executed directly from nonvolatile memory. For example, the display firmware 1040 may be embodied in a ROM chip coupled to the display 1012 such that the ROM chip is separate from other storage and system memory of the multiview display system 1000. The display 1012 may include processing circuitry for executing the display firmware 1040.
The operating system 1031, application(s) 1034, drivers (e.g., display driver 1037), firmware (e.g., display firmware 1040), and potentially other instruction sets may each comprise instructions that are executable by the processor(s) 1003 or other processing circuitry of the multiview display system 1000 to carry out the functionality and operations discussed above. Although the instructions described herein may be embodied in software or code executed by the processor(s) 1003 as discussed above, as an alternative, the instructions may also be embodied in dedicated hardware or a combination of software and dedicated hardware. For example, the functionality and operations carried out by the instructions discussed above may be implemented as a circuit or state machine that employs any one of or a combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field-programmable gate arrays (FPGAs), or other components, etc.
In some embodiments, the instructions that carry out the functionality and operations discussed above may be embodied in a non-transitory, computer-readable storage medium. The computer-readable storage medium may or may not be part of the multiview display system 1000. The instructions may include, for example, statements, code, or declarations that can be fetched from the computer-readable medium and executed by processing circuitry (e.g., the processor(s) 1003). In the context discussed herein, a ‘non-transitory, computer-readable medium’ may be any medium that can contain, store, or maintain the instructions described herein for use by or in connection with an instruction execution system, such as, for example, the multiview display system 1000.
The non-transitory, computer-readable medium can comprise any one of many physical media such as, for example, magnetic, optical, or semiconductor media. More specific examples of a suitable computer-readable medium may include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer-readable medium may be a random access memory (RAM) including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other type of memory device.
The multiview display system 1000 may perform any of the operations or implement the functionality descried above. For example, the flowchart and process flows discussed above may be performed by the multiview display system 1000 that executes instructions and processes data. While the multiview display system 1000 is shown as a single device, embodiments are not so limited. In some embodiments, the multiview display system 1000 may offload processing of instructions in a distributed manner such that a plurality of multiview display systems 1000 or other computing devices operate together to execute instructions that may be stored or loaded in a distributed arranged. For example, at least some instructions or data may be stored, loaded, or executed in a cloud-based system that operates in conjunction with the multiview display system 1000.
Thus, there have been described examples and embodiments of tilting a plane of convergence of a multiview image. For example, the plane of convergence may be tilted in a real-time graphics pipeline as the multiview image is rendered for display. In this respect, the plane of convergence may be tilted by applying a shear function to different views of the multiview image based on the relative location of each view. It should be understood that the above-described examples are merely illustrative of some of the many specific examples that represent the principles described herein. Clearly, those skilled in the art can readily devise numerous other arrangements without departing from the scope as defined by the following claims.
This application is a continuation patent application of and claims priority to International Patent Application No. PCT/US2020/066251, filed Dec. 18, 2020, which claims the benefit of priority to U.S. Provisional Application No. 63/115,531, filed on Nov. 18, 2020, the entirety of each of which are incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 63115531 | Nov 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2020/066251 | Dec 2020 | US |
| Child | 18197078 | US |
