Patent Application
20220124301
The invention relates to a system for displaying views of an application on a display, wherein the views represent 3D image content for a 3D display. The invention further relates to a computer-implemented method for providing, by the application, the 3D image content to a window manager, and to a computer-implemented method for determining, by a display server component, that the views represent 3D image content.
Increasingly, display devices such as televisions, digital photo frames, tablets and smartphones comprise 3D displays to provide a user with a perception of depth when viewing content on such a device. For that purpose, such 3D displays may, either by themselves or together with glasses worn by the user, provide the user with different images in each eye so as to provide the user a perception of depth based on stereoscopy.
The 3D image content to be displayed on such 3D displays may generally be represented by 2D image data and so-called 3D-enabling auxiliary data. The latter data may be indicative of depth of the 2D image data. For example, the 3D-enabling auxiliary data may be further 2D image data which together with the 2D image data represents a pair of stereo images, or depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer. Such depth-related data may contain depth values, but also disparity values, parallax values or other types of depth-related values.
The 3D image content may be displayed by an application running on an operating system. Such an operating system may provide a window manager for managing the visibility of views generated by applications. In addition, the operating system may provide one or more display server components which may each be specific to a type of display and which may be configured to, based on visibility information obtained from the window manager, composite the views of application(s) into a display signal for display. These display server components may be components of a single software component called ‘display server’, but may also each represent a separate display server.
It is noted that the window manager and display server(s) may be implemented as separate software components but may also be combined in a single software component or in another manner partitioned over software components. A specific example is that the Android operating system provides a window manager named ‘WindowManager’ and a display server named ‘SurfaceFlinger’. Various other types of window managers and/or display servers are known as well, for example, as part of other operating systems.
A disadvantage of known ways of displaying 3D image content by an application is that the application may be inflexible with respect to the type of display server component and display. Namely, a particular application may support only one type of display, e.g., a 2D or 3D display, or may in principle support different types of displays but may be unaware of the display type connected to the system and may need to be manually configured.
Such flexibility is desirable, for example to support dynamic switching, e.g., during runtime, between different types of displays or to support multi-display (also referred to as ‘multi-monitor’) setups involving different types of displays. Such type of flexibility may also be referred to as ‘interoperability’. In particular, it is desirable to provide such interoperability between 2D (‘legacy’) displays and 3D displays. Although in many cases a legacy 2D display may reproduce 3D image content which may be generated or output by a particular application, such 3D image content may then be reproduced in the form of a 2D image, which may be undesirable. For example, if the 3D image content is stereo content provided as two views side-by-side, a 2D display server component may render both views side-by-side on the 2D display. This is not desirable, as it rather may be desired to render only one of both views on a 2D display while disregarding the other view.
US 2012/092335 A1 describes a method and a 3D display apparatus for processing a stereoscopic image signal by software while using a least number of hardware components in a portable 3D display apparatus based on a mobile Android platform are provided. One or more plane image surfaces are generated from an application/middleware layer and stored in a first frame buffer. An encoded image signal is decoded under the application/middleware layer to restore a YUV image signal representing a stereoscopic image pair. Subsequently, the YUV image signal is converted into an RGB image signal, and left and right images of the RGB image signal are mixed at the kernel layer.
It is an object of the invention to provide a way of displaying 3D image content by an application which is more flexible with respect to the type of display server component and display.
In accordance with a first aspect of the invention, a system is provided for displaying views of an application on a display, wherein the views represent 3D image content for a 3D display. The system comprises:
In accordance with a further aspect of the invention, a display device is provided comprising the system.
In accordance with a further aspect of the invention, a computer-implemented method is provided for providing views of an application to a window manager of an operating system, wherein the views represent 3D image content for a 3D display, wherein the operating system is configured to provide:
the window manager, wherein the window manager is configured to manage visibility of views generated by applications;
one or more display server components, the one or more display server components being specific to a type of display and configured to, based on visibility information obtained from the window manager, composite the views into a display signal for display;
the method comprising, by the application, providing the 3D image content to the window manager in the form of at least two views which are arranged with respect to each other in accordance with a view configuration, wherein the at least two views comprise a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data, and wherein the view configuration causes a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal.
In accordance with a further aspect of the invention, a computer-implemented method is provided for compositing views of an application executed on an operating system, wherein the views represent 3D image content for a 3D display, wherein the operating system is configured to provide:
a window manager for managing visibility of views generated by applications;
a 3D display server component for a 3D display, the 3D display server component being configured to, based on visibility information obtained from the window manager, composite the views into a display signal for the 3D display;
wherein the application is configured to provide the 3D image content to the window manager in the form of at least two views which are arranged with respect to each other in accordance with a view configuration, wherein the at least two views comprise a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data, and wherein the view configuration causes a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal;
A further aspect of the invention provides a transitory or non-transitory computer-readable medium comprising a computer program representing the application or the 3D display server component, the computer program comprising instructions for causing a processor system to perform the method representing the respective entity.
The above measures involve an application providing the 3D image content to the window manager in the form of at least two views, namely a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data. Here and elsewhere, the term ‘view’ may refer to a basic building block of a graphical user interface, which may also be referred to as ‘view object’. Typically, a view represents a rectangular area on the screen which may display content of the application. For example, in Android, such a view may be represented by the public class ‘View’, while in iOS, such a view may be represented by a ‘View Object’, etc. It is noted that in many operating systems, a ‘window’ of an application may contain one or more views, and as such, a view may not need to represented by a separate window.
The primary view may thus show an image, such as a photograph (still image), a video frame, an image generated by computer rendering, etc., while the 3D-enabling auxiliary data may be indicative of the depth of said image. The concept of representing 3D image content by 2D image data and 3D-enabling auxiliary data is known per se. Such auxiliary data may take various forms, including but not limited to further 2D image data which together with the 2D image data represents a pair of stereo images, or depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer.
The application may thus create a separate view for the 2D image data and a separate view for the 3D-enabling auxiliary data. The views are mutually, e.g., with respect to each other in a spatial sense, arranged in accordance with a view configuration. Such a view configuration may be defined and/or provided by the application and may be represented by one or more parameters which define at least a relative position of both views, for example, within the application's window. For example, the view configuration may be a predetermined view configuration. For example, in Android, the parameters may be parameters of the public class ‘RelativeLayout’ according to which the positions of views can be described in relation to each other. The thus-configured relative position may then be provided by the application using the function Set.ContentView having the particular RelativeLayout as argument.
It will be appreciated that depending on the type of operating system and/or window manager, various other mechanisms exist and may be advantageously used to define and/or provide a view configuration for the primary view and the secondary view.
In accordance with the above measures, the application specifically defines and/or provides the view configuration such that a 2D display server component for a 2D display, e.g., a ‘legacy’ display server component, omits drawing or overdraws the secondary view when compositing the at least two views into the display signal. As such, the 2D display server component will omit displaying the 3D-enabling auxiliary data, and rather (only) display the 2D image data. Here and in the following, the term ‘display signal’ may refer to a signal formatted in a display-specific manner by the display server component, which in itself may be known per se. The display signal may for example be generated as an output image in an output buffer which is formatted in a display-specific manner.
A non-limiting example is that the application may be configured to stack the primary view in front of the secondary view to provide as the view configuration a view configuration in which the primary view occludes the secondary view. Such stacking of views may in fact be a default behavior of the public class ‘RelativeLayout’, with the application only having to ensure that the primary view is stacked in front of the secondary view, rather than the other way around. This may, for example, be done by the application assigning a relative Z-order to both views which specifically causes the primary view to be stacked in front of the secondary view. Again referring to the example of the public class ‘RelativeLayout’, this may involve the application or programmer choosing an appropriate enumeration order of the views, or using the function view.setZ(float), etc.
A 2D display server component, if present and active in the operating system may then overdraw the secondary view, or may omit drawing the secondary view when the component determines that the secondary view is occluded by the primary view.
At the same time, a 3D display server component, if present and activate in the operating system, may determine that the at least two views represent 3D image content, for example on the basis of signaling received from the application, or metadata of the application. Having identified that the primary view and the secondary view represent 3D image content, the 3D display server component may then process the primary view and the secondary view in a manner as known per se so as to obtain the display signal for the 3D display. As such, a 3D display server component may not overdraw the secondary view, but may rather, for example, place the secondary view side-by-side besides the primary view in the display signal.
The above measures provide a backward-compatible way of outputting 3D image content by an application. Namely, a ‘legacy’ 2D display server component may show the primary view, while a 3D display server component may access the 3D-enabling auxiliary data in the secondary view and process the 2D image data and the 3D-enabling auxiliary data appropriately.
This may not only provide backwards compatibility to 2D display server components, but also to other (‘legacy’) types of display server components which are not configured to determine that the at least two views provided by the application represent 3D image content. Such legacy display server components may include the aforementioned 2D display server components, but also legacy 3D display server components and streaming or casting servers such as Miracast. Yet another advantage is that the application may not have to detect which type of display server component is present and active and adjust its output accordingly. The latter may not even be possible in a multi-display context in which different types of display server components are present and active.
It is noted that in the above and following, any references to the operating system being configured to ‘establish’ or ‘provide’ the window manager and/or the one or more display server components may refer to the operating system being configured to allow said software components to be executed by or using the operating system. In some embodiments, the operating system may comprise the window manager and/or the display server(s). In other embodiments, the window manager and/or the display server(s) may be software components which may be provided separately from the operating system.
The application may be configured to stack the primary view in front of the secondary view to provide as the view configuration a view configuration in which the primary view occludes the secondary view. For example, the application may be configured to assign a relative Z-order to the primary view and the secondary view which causes the primary view to be stacked in front of the secondary view.
Optionally, the application is configured to provide a barrier view stacked in between the primary view and the secondary view, wherein the barrier view is opaque and comprises homogenous image data. It may be the case that the primary view is defined to be (partially) translucent, or the primary view may be treated by a display server component as being (partially) translucent. For example, the 2D image data of a primary view may contain RGBA tuples, which are known per se, to provide local transparencies. As such, while a 2D display server component may still overdraw the secondary view with the primary view, the (partial) translucency of the primary view may cause the secondary view to still be (partly) visible, which may disturb the display of the 2D image data in the primary view. To avoid or reduce such disturbances, a barrier view may be provided stacked in between the primary view and the secondary view, for example by selecting a Z-order in between the Z-order of the primary view and the Z-order of the secondary view. The barrier view is generated by the application to be opaque, e.g., non-translucent, so as to cause the secondary view to be occluded by the barrier view during view composition by the 2D display server component. The barrier view is further generated to comprise homogenous image data, so as to reduce any disturbance to the 2D image data in the overlaying primary view. For example, the barrier view may be homogenously black, dark grey or light grey.
The application may be configured to:
Various alternatives exist to the stacking the primary view in front of the secondary view. For example, the primary view may be provided to the window manager by indicating a viewport to the window manager which comprises the 2D image data of the primary view, e.g., in a buffer. The 3D-enabling auxiliary data may be arranged outside of the viewport, e.g., in the same buffer outside of the viewport, or in a different buffer. Accordingly, a ‘legacy’ display server component may draw the primary view as shown in the viewport, but may omit drawing the secondary view as it is located outside of the viewport.
Optionally, the one or more display server components comprise a 3D display server component for a 3D display, and the application is configured to signal the 3D display server component that the at least two views represent 3D image content. If a 3D display server component for a 3D display is present and active in the operating system, the application may signal the 3D display server component that the two views represent 3D image content. For example, the 3D display server component may provide an Application Programming Interface (API) for allowing applications to interface with the 3D display server component, and the application may be configured to signal the 3D display server component via the API that the at least two views represent 3D image content, for example by registering an identifier of the at least two views with the 3D display server component via the API. Additionally or alternatively, the 3D display server component may be configured to detect that the at least two views represent 3D image content based on metadata of the application, such as an identifier of the application. As such, the 3D display server component may learn that the at least two views represent 3D image content and process the image(data) in the two views accordingly.
Optionally, the 3D display server component is configured to composite the at least two views by 1) arranging the primary view and the secondary view to be simultaneously shown in the display signal in accordance with a stereo display format, or 2) generating one or more further views based on the primary view and the secondary view, and simultaneously arranging the primary view and the one or more further views in the display signal in accordance with a multiview display format. The above options define ways which are known per se for a 3D display server component to process the primary view and the secondary view to obtain the display signal for the 3D display. The first option may be applicable to stereo content (e.g., 2D image data represents left image, 3D-enabling auxiliary data represents right image), in which the view composition may involve showing both views simultaneously, e.g., side-by-side, in accordance with a top-bottom stereo format or spatially interleaved, e.g., on a line basis. The second option may be applicable to so-called multiview content in which the 3D-enabling auxiliary data represents depth-related content, such as a depth map, and in which so-called view-rendering or view-synthesis techniques are used to render one or more further views representing alternative viewpoints of a scene besides the one shown in the primary view. It is noted that conversions between the different types of 3D display formats are known. As such, irrespective of the type of 3D-enabling auxiliary content, the views may be processed to obtain a desired display format. For example, if the application outputs stereo content, a 3D display server component may estimate disparity between both images and use the disparity map to render one or more further views for a multiview display. Conversely, if the application provides a depth map as 3D-enabling auxiliary content, a display server component may view-render a second image to provide stereo content as output.
It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and/or optional aspects of the invention may be combined in any way deemed useful.
Modifications and variations of any computer-implemented method and/or any computer program product, which correspond to the described modifications and variations of a corresponding system, can be carried out by a person skilled in the art on the basis of the present description, and vice versa.
These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,
It should be noted that items which have the same reference numbers in different Figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.
The following list of reference numbers is provided for facilitating the interpretation of the drawings and shall not be construed as limiting the claims.
As can be seen in
The application 200 may then generate a 2D view 400 which comprises the 2D image data 300. The 2D view 400 is also referred to as ‘primary view’. Here, the term ‘comprises’ refers to the 2D view 400 normally showing the 2D image data 300, albeit subject to visibility limitations of the 2D view 400 which may, for example, be imposed by the window manager 220. The application 200 may further generate a 3D-enabling auxiliary view 410 which comprises the 3D-enabling auxiliary data. The 3D-enabling auxiliary view 410 is also referred to as ‘secondary view’. The data of both views 400, 410 may, for example, be stored in respective buffers 210, 212, for example as buffer objects, or in a same buffer as separate buffer objects (not shown in
The primary view 400 and the secondary view 410 are mutually arranged by the application in accordance with a view configuration which causes a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal. There are various ways to define or provide such a view configuration, as e.g. elucidated with reference to
This concept may also be referred to as ‘explicit overdraw’ in which an output image is updated multiple times. A black or otherwise opaque and homogenous barrier view may be inserted in between the primary view 400 and the secondary view 410 in case the primary view 400 is (semi) transparent. Here, “inserted in between” refers to the barrier view having a Z-order placing it behind the primary view 400 but before the secondary view 410, with the barrier view preferably having a same location and size as the primary view 400 and/or the secondary view 410. Alternatively, if the display server component is able to determine that the secondary view 410 is occluded, the display server component may also entirely omit drawing the secondary view 410.
It is noted that to cause the overdrawing of ‘legacy’ display server components, the views 400, 410 generated by the application may be of a type that can only be composed by the operating system in order to bypass app composition optimization. For example, in Android, this may be achieved by the views 400, 410 being ‘SurfaceView’ type of views.
With continued reference to
The 3D display server component 254 may then create a mosaic-like composition of the 2D image data of the primary view 400, labeled ‘2D’ in
It is noted that also various other known types of multiview compositions may be used instead of a spatial mosaic-like composition. For example, the views may be spatially and/or temporally interleaved. Also other data may be provided in the display signal in addition to or instead of the viewpoints. For example, the 3D-enabling auxiliary data itself may be provided in the display signal, and/or transparency information and/or occlusion information. Another example is that the display signal may comprise a point cloud or a 3D mesh, etc., which are generated from the at least two views. It is noted that in general, the display signal may be provided to the display over a single but also over multiple independent data channels and/or and cables, e.g. via dual DisplayPort.
According to a second example, the 3D display server component 256 may be configured to generate a display signal 444 for a stereo display. This view composition may take different forms depending on the type of 3D-enabling auxiliary data 310. For example, the 3D-enabling auxiliary data 310 represents further 2D image data which together with the 2D image data represents a pair of stereo images. In this case, if the 2D image data 300 represents a left image and the further 2D image data 310 represents a right image, the 3D display server component 256 may composite them into a side-by-side formatted display signal 444. It is noted that also various other known types of spatial and/or temporal stereo view compositions may be used instead. If, however, the 3D-enabling auxiliary data 310 represents depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer, the further 2D image data (e.g., the right image) may be generated using the aforementioned view-rendering or view-synthesis techniques.
In the example of
There are various ways to enable the 3D display server component 264 to determine that the stacked views 400, 410 of the application 200 represent 3D image content. In the example of
Although described with reference to Android, the above measures may be applied to various other types of operating systems by a person skilled in the art of application development for the particular operating system. For example, for *nix-based operating systems (Linux, BSD, Unix, etc.), a wide variety of window managers exist (see https://wiki.archlinux.org/index.php/window_manager) as well as various customizable display servers (e.g., X11, Wayland-based, Mir, DirectFb). For MacOS or iOS (Quartz), the Quartz Window Services may provide at least part of the functionality of the window server as described, and the Quartz Display Services/Quartz Composition Services/XQuartz may configured to provide at least part of the functionality of the display server of display server components as described. For Microsoft Windows, the Desktop Window Manager may provide at least part of the functionality of the window server as described, and the GDI Graphics Device Interface may be configured to provide at least part of the functionality of the display server or display server components as described.
In general, the system may be embodied in or as a separate device, e.g., in or as a set-top box, personal computer, gaming console or similar device that is connectable to a (3D) display. Alternatively, the system may be embodied in or as a display device which comprises the (3D) display, e.g., in or as a smartphone, tablet device, television, display, monitor, etc. In general, the system may be implemented by a device or apparatus. The device or apparatus may comprise one or more (micro) processors which execute appropriate software. Software implementing the functionality of the function(s) may have been downloaded and/or stored in a corresponding memory or memories, e.g., in volatile memory such as RAM or in non-volatile memory such as Flash. Alternatively, the function(s) of the system may be implemented in the device or apparatus in the form of programmable logic, e.g., as a Field-Programmable Gate Array (FPGA), or as an Application-Specific Integrated Circuit (ASIC), or as any other type of circuit or combination of circuits. Any of the software components described in this specification may be represented by instructions for a computer, e.g., executable code, which may be stored on a computer readable medium 500, e.g., in the form of a series 510 of machine readable physical marks and/or as a series of elements having different electrical, e.g., magnetic, or optical properties or values. The executable code may be stored in a transitory or non-transitory manner. Examples of computer readable mediums include memory devices, optical storage devices, online software, etc.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of” when preceding a list or group of elements represent a selection of all or of any subset of elements from the list or group. For example, the expression, “at least one of A, B, and C” should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
| Number | Date | Country | Kind |
|---|---|---|---|
| 19153263.9 | Jan 2019 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/051387 | 1/21/2020 | WO | 00 |
