Patent Grant
11854172
This is a U.S. national stage application of PCT Application No. PCT/CN2021/070529 under 35 U.S.C. 371, filed Jan. 6, 2021 in Chinese, claiming priority to Chinese Patent Applications No. 202010181920.3, filed Mar. 16, 2020, all of which are hereby incorporated by reference.
The present invention belongs to the field of real-time rendering, and in particular, relates to a color contrast enhanced rendering method, device and system suitable for an optical see-through head-mounted display.
With the continuous innovation of optical see-through head-mounted displays (hereinafter OST-HMD for short), the related mixed reality (hereinafter MR for short) is also developed continuously. Different from the conventional virtual reality head-mounted display or video see-through head-mounted display, the OST-HMD has a semitransparent optical lens, so the OST-HMD may present real and non-drawn ambient environments and virtual and drawn display contents at the same time. This design enables a user to perceive the surrounding environment without delay and with high fidelity when wearing the OST-HMD, so that the discomfort of the user who wears the video see-through head-mounted display is greatly alleviated, but new problems ensued. The optical structure of the OST-HMD makes the OST-HMD unable to completely block light from the external environment, so that the virtual drawn content cannot be independently presented on the display. In the MR application scene, the optical design will lead to the so-called color-blending problem, that is, since the virtual display content and the color of the real background environment are blended, the virtual content becomes difficult to see clearly, especially when there is a low color contrast between the virtual content and the background environment.
Under such a background, it has become a practical demand to enhance the discrimination between the virtual display content and the real background environment. However, there is no universal solution suitable for the mainstream commercial OST-HMD at present, and there are many possibilities to be explored in this direction. In the related strategies to relieve the color-blending problem on the OST-HMD, controlling the opaqueness of a display pixel by increasing an additional hardware system has been proved to be an effective hardware solution. In such a solution, complete or partial blocking effect between the virtual display content and the real background environment may be achieved by adjusting the opaqueness of the display pixel, so that the discrimination between the virtual display content and the real background environment is improved. However, this hardware solution usually does not have scalability and is difficult to realize the miniaturization of equipment overall dimension, and the production cost is high.
In addition to the hardware solution, some software solutions perform accurate pixel-level matching on the display picture and the background environment to subtract the color component of the real background environment from the virtual display content, so that the real background environment corresponding to the mixed virtual display content is almost invisible and the discrimination between the display picture and the background environment is finally improved. However, the solution for the mainstream commercial OST-HMD is adopted in the daily environment to reduce the brightness of the virtual display content, so that the transparency of the virtual display content is increased, and the discrimination between the virtual display content and the real background environment is reduced.
However, there is also a software solution for the mainstream commercial OST-HMD to increase the opaqueness of the virtual display content by enhancing the brightness of the virtual display content, so that the discrimination between the virtual display content and the real background environment is improved. However, this solution will reduce the contrast of the virtual display content itself and lose the specific details, so this solution is not suitable for the virtual content with complex texture and has low universality.
An object of the present invention is to provide a color contrast enhanced rendering method, device and system suitable for an optical see-through head-mounted display. According to the color contrast enhanced rendering method, an optimal rendering color based on an original rendering color and a background environment color is found pixel by pixel under the condition of meeting a series of constraint conditions.
The technical solution of the present invention is as follows:
A color contrast enhanced rendering method suitable for an optical see-through head-mounted display comprises the following steps:
The original rendering color refers to an original color displayed by the OST-HMD on its display screen before this technical solution is applied; the original rendering color space refers to a color space adopted by the original rendering color; the background video color refers to a color of a video of a background environment behind the OST-HMD acquired by video acquisition equipment (such as a camera); the background video color space refers to a color space adopted by the background video color; and converting the original rendering color and the processed background video color from their own color spaces to the CIELAB color space refers to that the original rendering color is converted from the original rendering color space to the CIELAB color space and the background video color is converted from the video color space to the CIELAB color space. The original rendering color space may be a color space such as RGB, HSV, YUV, CIEXYZ and CIELAB, and the video color space may be a color space such as RGB, HSV, YUV and YCbCr.
Preferably, the step of pre-processing the background video comprises:
performing Gaussian blur processing on the background video, specifically, filtering the background video by using a Gaussian filter core to obtain a blur effect, wherein the mathematical description of the Gaussian filter core is as follows:
wherein x and y respectively represent horizontal and vertical distances from the pixel to a center pixel, and σ represents a standard deviation of the selected Gaussian distribution;
performing visual field correction on the background video subjected to Gaussian processing to realize pixel accuracy one-to-one mapping of a frame image of the background video and a rendering picture, wherein a correction formula is:
wherein u and v respectively represent texture coordinates of the rendering picture in horizontal and vertical directions, x and y respectively represent texture coordinates of the frame image of the background video in the horizontal and vertical directions, su and sv respectively represent scaling coefficients of the texture coordinates of the frame image of the background video in the horizontal and vertical directions, and bu and bv respectively represent offsets of the texture coordinates of the frame image of the background video in the horizontal and vertical directions.
The step (2) specifically includes:
(2-1) converting the original rendering color and the processed background video color from their own original color spaces to the CIELAB color space; and
(2-2) scaling the three-dimensional coordinates of the original rendering color and the background video color in the CIELAB color space to a unit sphere range, that is, an Euclidean distance from the three-dimensional coordinates corresponding to any color in the CIELAB color space after scaling to the origin is less than or equal to 1.
In the step (3),
the color difference constraint means that a color difference of the optimal rendering color and the original rendering color should be kept within a certain range, and the color different constraint is defined as follows:
ΔE*ab(lopt,ld)≤λE
wherein ΔE*ab(•) represents the color difference of the two colors in the CIELAB color space, lopt and ld respectively represent the optimal rendering color and the original rendering color, and λE represents a color difference threshold with a positive value;
the chromaticity saturation constraint means that a chromaticity saturation of the optimal rendering color should not be reduced, and the chromaticity saturation constraint is defined as follows:
chopt−chd≥0
wherein chopt and chd respectively represent chromaticity saturations of the optimal rendering color and the original rendering color in the CIELAB color space;
the brightness constraint means that a brightness difference of the optimal rendering color and the original rendering color should be kept within a certain range, and the brightness constraint is defined as follows:
ΔL*(lopt,ld)≤λL
wherein ΔL*(•) represents the brightness difference of the two colors in the CIELAB color space, and λL represents a brightness threshold with a positive value; and
the just noticeable difference constraint means that the optimal rendering color and the background video color should have a color difference distinguishable by human eyes, and the just noticeable difference constraint is defined as follows:
ΔE*ab(lopt,ld)≥λJND
wherein lb represents the background video color, and λJND represents a just noticeable difference with a positive value.
The step (3) step specifically comprises:
(3-1) calculating an ideal optimal rendering color by using the following formula:
wherein I and B respectively represent three-dimensional coordinates of the ideal optimal rendering color and the background video color subjected to Gaussian blur in the CIELAB color space which is scaled to a unit sphere, O represents an origin of coordinates and, O is also a center of the unit sphere, dist(•) represents an Euclidean distance between two points in the space, norm(•) represents that a vector is normalized, {right arrow over (OB)} represents a vector with a starting point at O and an end point at B; and
(3-2) applying the color difference constraint to the ideal optimal rendering color, wherein the method for applying the color difference constraint is as follows:
{right arrow over (DE)}=min(dist(D,I),λ′E)·norm({right arrow over (DI)})
wherein D and E respectively represent three-dimensional coordinates of the original rendering color and the ideal optimal rendering color to which the color difference constraint is applied in the CIELAB color range scaled to the unit sphere, min(•) represents taking a smaller value of the two three-dimensional coordinates, λ′E represents a color different threshold scaled to the unit sphere range, and {right arrow over (DI)} represents a three-dimensional vector with a starting point at D and an end point at I;
(3-3) continuing to apply the chromaticity saturation constraint to the ideal optimal rendering color to which the color difference constraint is applied, and making a superscript ′ represent taking a projection vector of a vector on a a*Ob* plane in the CIELAB color space, wherein the method for applying the chromaticity saturation constraint is as follows:
wherein {right arrow over (DC)} represents a {right arrow over (DE)}′ vector after the chromaticity saturation constraint is applied, DE′ch represents a vector component of a {right arrow over (DE)}′ vector in a {right arrow over (OD)}′ vector direction, that is, a variation vector of the chromaticity saturation constraint, {right arrow over (DE)}′h represents another vector component of the {right arrow over (DE)}′ in a direction vertical to the {right arrow over (DE)}′ch, vector, that is, {right arrow over (DE)}′h={right arrow over (DE)}′−{right arrow over (DE)}′ch, and θch represents an angle from the {right arrow over (OD)}′ vector to the {right arrow over (DE)}′ vector;
(3-4) continuing to apply the brightness constraint to the ideal optimal rendering color to which the color different constraint and the chromaticity saturation constraint are applied, wherein the method for applying the brightness constraint is as follows:
{right arrow over (DL)}=(1−|cos θl|)·{right arrow over (DE)}L*
wherein {right arrow over (DE)}L* represents a vector component of a {right arrow over (DE)} vector on a L* axis in the CIELAB color space scaled to the unit sphere, {right arrow over (DL)} represents a {right arrow over (DE)}L* vector after the brightness constraint is applied, θl represents an angle from a positive direction of the L* axis to the {right arrow over (DE)} vector; and
(3-5) continuing to apply the just noticeable difference constraint to the ideal optimal rendering color to which the color different constraint, the chromaticity saturation constraint and the brightness constraint are applied, wherein the method for applying the just noticeable difference constraint is as follows:
{right arrow over (DP)}={right arrow over (DC)}+{right arrow over (DL)}
Lopt=D+r·{right arrow over (DP)}
wherein Lopt represents three-dimensional coordinates of the optimal rendering color in the CIELAB color space scaled to the unit sphere, and r represents a scaling coefficient: when an Euclidean distance from P to B is less than the just noticeable difference scaled to the unit sphere range, r is less than 1, so as to reduce a module length of a {right arrow over (DP)} vector with a starting point at D and an end point at P, so that an Euclidean distance from the end point of the reduced {right arrow over (DP)} vector to B is not less than the just noticeable difference scaled to the unit sphere range, and if an Euclidean distance from the P to the B is already greater than or equal to the just noticeable difference scaled to the unit sphere unit, r is equal to 1.
The step (4) specifically comprises:
(4-1) scaling the optimal rendering color in the CIELAB color space scaled to the unit sphere range back to a coordinate range of the original CIELAB color space;
(4-2) converting the optimal rendering color in the CIELAB color space back to the original rendering color space; and
(4-3) replacing the original rendering color with the converted optimal rendering color for real-time rendering.
In the present invention, the background environment may be acquired in real time by a video camera of the OST-HMD, only the color difference threshold λE and the just noticeable difference λJND need to be determined by a user, and other parameters such as the brightness threshold λL are automatically set by the present invention during operation.
A color contrast enhanced rendering device suitable for an optical see-through head-mounted display comprises:
an acquisition and processing module, configured to acquire a background environment in real time to obtain a background video and perform Gaussian blur and visual field correction on the background video to obtain a processed background video;
a color space conversion module, configured to convert the original rendering color and the processed background video color from their own original color spaces to a CIELAB color space, and further configured to convert the optimal rendering color from the CIELAB color space to the original rendering color space;
an optimal rendering color calculation module, configured to find an optimal rendering color based on the original rendering color and the processed video color in the CIELAB space according to a set color difference constraint, a chromaticity saturation constraint, a brightness constraint and a just noticeable difference constraint; and
a rendering module, configured to replace the original rendering color by the converted optimal rendering color for real-time rendering.
A color contrast enhanced rendering system suitable for an optical see-through head-mounted display processor comprises a memory and a computer program which is stored in the processor and capable of being executed by the processor, wherein the computer program, when being executed by the processor, implements the above color contrast enhanced rendering method suitable for the optical see-through head-mounted display.
Compared with the prior art, the present invention has the following beneficial effects:
the application range is wide and is not limited to a certain specific application program or certain specific hardware; the rendering method of the present invention is a full-screen post-processing effect, may be applied to rendering frameworks such as Vulkan, DirectX and OpenGL and other commercial game engines, and may also be popularized to various different OST-HMDs;
within the color difference threshold given by a user, by increasing the color contrast between the virtual display content and the real background environment, the discrimination between the virtual display content and the real background environment is improved on the premise of ensuring the presentation of the virtual content as much as possible, and the visibility reduction or detail loss of the virtual display content caused by the prior art is avoided; and
the rendering method supports various mixed reality application scenes without any pre-preparation process and may be applied to various common background environments.
In order to describe the technical solutions in the embodiments of the present invention or in the prior art more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments or the prior art. Apparently, the accompanying drawings in the following description show merely some embodiments of the present invention, and a person of ordinary skill in the art may also derive other drawings from these accompanying drawings without creative efforts.
In order to make the objectives, technical schemes and advantages of the present invention more clearly, the present invention will be further described below in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the protection scope of the present invention.
Referring to
S01: a background environment is acquired in real time by a video camera of the optical see-through head-mounted display to obtain a background video.
In this embodiment, a resolution ratio of the generated background video was 768×432, and a frame rate is 24 frames/second.
S02: the background video was filtered by using a Gaussian filter core to obtain a blur effect.
In this embodiment, the mathematical description of the Gaussian filter core required to perform Gaussian filtering is as follows:
wherein x and y respectively represent horizontal and vertical distances from the pixel to a center pixel, and σ represents a standard deviation of the selected Gaussian distribution. In this embodiment, a standard deviation of the selected Gaussian distribution was 3.
S03: the background video was subjected to visual field correction by using a visual field correction formula.
In this embodiment, the visual field correction was defined as follows:
wherein u and v respectively represent texture coordinates of the rendering picture in horizontal and vertical directions, x and y respectively represent texture coordinates of the frame image of the background video in the horizontal and vertical directions, su and sv respectively represent scaling coefficients of the texture coordinates of the frame image of the background video in the horizontal and vertical directions, and bu and bv respectively represent offsets of the texture coordinates of the frame image of the background video in the horizontal and vertical directions. In this embodiment, the values of both su and sv were 0.65, and the values of bu and bv were 0.13 and 0.17 respectively.
S04: the original rendering color and the background video color were converted from an RGB color space to an CIELAB color space pixel by pixel, the three-dimensional coordinates of the original rendering color and the background video color in the CIELAB color space were scaled to a unit sphere range, that is, an Euclidean distance from the three-dimensional coordinates corresponding to any color in the CIELAB color space after scaling to an origin was less than or equal to 1.
S05: according to a color different threshold given by a user, an optimal rendering color based on the original display color and the background video color was found pixel by pixel under the condition of meeting a series of constraint conditions.
In this embodiment, there were four constraint conditions, which were respectively defined as follows:
(a) a color difference constraint means that a color difference of the optimal rendering color and the original rendering color should be kept within a certain range, and the color different constraint was defined as follows:
ΔE*ab(lopt,ld)≤λE
wherein ΔE*ab(•) represents the color difference of the two colors in the CIELAB color space, lopt and ld respectively represent the optimal rendering color and the original rendering color, and λE represents a color difference threshold with a positive value;
(b) a chromaticity saturation constraint means that a chromaticity saturation of the optimal rendering color should not be reduced, and the chromaticity saturation constraint was defined as follows:
chopt−chd≥0
wherein chopt and chd respectively represent chromaticity saturations of the optimal rendering color and the original rendering color in the CIELAB color space;
(c) a brightness constraint means that a brightness difference of the optimal rendering color and the original rendering color should be kept within a certain range, and the brightness constraint was defined as follows:
ΔL*(lopt,ld)≤λL
wherein ΔL*(•) represents the brightness difference of the two colors in the CIELAB color space, and λL represents a brightness threshold with a positive value; and
(d) a just noticeable difference constraint means that the optimal rendering color and the background video color should have a color difference distinguishable by human eyes, and the just noticeable difference constraint was defined as follows:
ΔE*ab(lopt,lb)≥λJND
wherein lb represents the background video color, and λJND represents a just noticeable difference with a positive value.
The S05 specifically comprises:
S05-1: an ideal optimal rendering color was calculated. According to this embodiment, the ideal optimal rendering color was calculated by using the following formula:
wherein I and B respectively represent three-dimensional coordinates of the ideal optimal rendering color and the background video color subjected to Gaussian blur in the CIELAB color space which was scaled to a unit sphere, O represents an origin of coordinates and, O is also a center of the unit sphere, dist(•) represents an Euclidean distance between two points in the space, norm(•) represents that a vector is normalized, {right arrow over (OB)} represents a vector with a starting point at O and an end point at B; and
S05-2: the color difference constraint was applied to the ideal optimal rendering color. The method for applying the color difference constraint was as follows:
{right arrow over (DE)}=min(dist(D,I),λ′E)·norm({right arrow over (DI)})
wherein D and E respectively represent three-dimensional coordinates of the original rendering color and the ideal optimal rendering color to which the color difference constraint was applied in the CIELAB color range scaled to the unit sphere, min(•) represents taking a smaller value of the two three-dimensional coordinates, λ′E represents a color different threshold scaled to the unit sphere range, and {right arrow over (DI)} represents a three-dimensional vector with a starting point at D and an end point at I;
S05-3: the chromaticity saturation constraint was continuously applied to the ideal optimal rendering color to which the color difference constraint was applied. A superscript ′ is made to represent taking a projection vector of a vector on a a*Ob* plane in the CIELAB color space. The method for applying the chromaticity saturation constraint was as follows:
wherein {right arrow over (DC)} represents a {right arrow over (DE)}′ vector after the chromaticity saturation constraint was applied, {right arrow over (DE)}′ch represents a vector component of a {right arrow over (DE)}′ vector in a {right arrow over (OD)}′ vector direction, that is, a variation vector of the chromaticity saturation constraint, {right arrow over (DE)}′h represents another vector component of the {right arrow over (DE)}′ in a direction vertical to the {right arrow over (DE)}′ch vector, that is, DE′h={right arrow over (DE)}′−{right arrow over (DE)}′ch, and θch represents an angle from the {right arrow over (OD)}′ vector to the {right arrow over (DE)}′ vector;
S05-4: the brightness constraint was continuously applied to the ideal optimal rendering color to which the color different constraint and the chromaticity saturation constraint were applied. The method for applying the brightness constraint was as follows:
{right arrow over (DL)}=(1−|cos θl|)·{right arrow over (DE)}L*
wherein {right arrow over (DE)}L* represents a vector component of a {right arrow over (DE)} vector on a L* axis in the CIELAB color space scaled to the unit sphere, {right arrow over (DL)} represents a {right arrow over (DE)}L* vector after the brightness constraint was applied, θl represents an angle from a positive direction of the L* axis to the {right arrow over (DE)} vector.
S05-5: the just noticeable difference constraint was continuously applied to the ideal optimal rendering color to which the color different constraint, the chromaticity saturation constraint and the brightness constraint were applied. In this embodiment, the method for applying the just noticeable difference constraint was as follows:
{right arrow over (DP)}={right arrow over (DC)}+{right arrow over (DL)}
Lopt=D+r·{right arrow over (DP)}
wherein Lopt represents three-dimensional coordinates of the optimal rendering color in the CIELAB color space scaled to the unit sphere, and r represents a scaling coefficient:
when an Euclidean distance from P to B was less than the just noticeable difference scaled to the unit sphere range, r was less than 1, so that a module length of a {right arrow over (DP)} vector with a starting point at D and an end point at P was reduced, and an Euclidean distance from the end point of the reduced {right arrow over (DP)} vector to B was not less than the just noticeable difference scaled to the unit sphere range, and if an Euclidean distance from the P to the B was already greater than or equal to the just noticeable difference scaled to the unit sphere unit, r was equal to 1. In this embodiment, the value of the just noticeable difference before scaling was 2.3.
S06: the optimal rendering color was converted from the CIELAB color space to a CIE XYZ color space pixel by pixel, and the optimal rendering color in the CIE XYZ color space was converted back to the RGB color space.
S07: the original rendering color was replaced by the optimal rendering color in the RGB color space for real-time rendering.
The experimental simulation result was as shown in Table 1 when the virtual display content was drawn by the method of this embodiment, wherein λ′E represents a color difference threshold given by a user. It may be seen that by the method of the present invention, a color contrast between a rendering color and a background color may be enhanced, so that a discrimination between the virtual content and the real background environment was obviously improved.
An embodiment further provides a color contrast enhanced rendering device suitable for an optical see-through head-mounted display, including:
It should be noted that when the color contrast enhanced rendering device suitable for the optical see-through head-mounted display provided by the above embodiment performs color contrast enhanced rendering, division of the foregoing function modules should be used as an example for description, and the foregoing functions may be allocated to and completed by different function modules as required. That is, an internal structure of a server was divided into different function modules, to complete all or some of the functions described above. That is, an internal structure of a terminal or a server was divided into different function modules to complete all or some of the functions described above. In addition, the color contrast enhanced rendering device suitable for the optical see-through head-mounted display provided by the above embodiment belongs to the same concept as the color contrast enhanced rendering method suitable for the optical see-through head-mounted display provided by the above embodiment. The details of the specific implementation process of the color contrast enhanced rendering device suitable for the optical see-through head-mounted display were shown in the embodiment of the color contrast enhanced rendering method suitable for the optical see-through head-mounted display, which are not elaborated herein.
An embodiment further provides a color contrast enhanced rendering system suitable for an optical see-through head-mounted display, including a processor, a memory and a computer program which was stored in the processor and capable of being executed by the processor, wherein the computer program, when being executed by the processor, implements the above color contrast enhanced rendering method suitable for the optical see-through head-mounted display.
The memory may include one or more computer readable storage mediums that may be non-transitory. The memory may further include a high-speed random access memory and a nonvolatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer readable storage mediums in the memory were used for storing at least one instruction which is used for being executed by the processor to implement the color contrast enhanced rendering method suitable for the optical see-through head-mounted display provided by the method embodiments of the present application.
The above specific embodiments have described the technical solutions and beneficial effects of the present invention in detail. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modification, supplement and equivalent substitution made within the principle scope of the present invention should be included in the protection scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202010181920.3 | Mar 2020 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2021/070529 | 1/6/2021 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2021/184931 | 9/23/2021 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20170323438 | Honeck et al. | Nov 2017 | A1 |
| 20190005652 | Pham | Jan 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 103581571 | Feb 2014 | CN |
| 108806600 | Nov 2018 | CN |
| 110419061 | Nov 2019 | CN |
| Number | Date | Country | |
|---|---|---|---|
| 20220261969 A1 | Aug 2022 | US |
