Patent Application
20200035027
This application claims priority to Korean Patent Application No. 10-2018-0088674, filed on Jul. 30, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.
Example embodiments consistent with the present disclosure relates to three-dimensional (3D) image display device and method.
Recently, 3D image display technology has been applied to various fields, including image display apparatuses related to a virtual reality (VR) display and an augmented reality (AR) display.
For example, a head mount display that provides VR display is currently in the commercialization stage and is widely applied to the entertainment industry. In addition, the head mount display is developing into a form that may be applied to medical, education and industrial fields.
An AR display, which is an advanced form of the VR display, is an image display apparatus that combines the real world and VR and has a characteristic that may lead to the interaction between reality and virtuality. The interaction between the real world and VR is based on the ability to provide real-time information about a real situation and may further increase an effect of reality by overlapping a virtual object or information on a real world environment.
In the related art AR display apparatuses and VR display apparatuses, a stereoscopy technique is usually used for the 3D image display, where visual fatigue may be involved due to a vergence-accommodation mismatch. Accordingly, a 3D image display method capable of reducing visual fatigue has been researched.
Provided are 3D image display apparatuses and methods capable of reducing visual fatigue.
Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the embodiments of the disclosure.
According to an aspect of the disclosure, there is provided a three-dimensional (3D) image display device comprising: a display device configured to modulate light according to 3D image information to form a 3D image; a variable focus optical system configured to focus the 3D image formed by the display device on a reference plane; a processor configured to determine a representative depth value of the 3D image by selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, as the representative depth value, and control the variable focus optical system to adjust the reference plane by adjusting a focal point of the variable focus optical system based on the representative depth value; and a transfer optical system configured to transfer the 3D image focused on the reference plane to a pupil of an observer.
The processor maybe further configured to determine the representative depth value based on a depth map and a color map of the 3D image.
The processor maybe further configured to quantize depth values included in the depth map into a plurality of quantized values and select at least one of the plurality of quantized values as the representative depth value.
The processor maybe further configured to determine the representative depth value based on saliency information from the color map.
The processor maybe further configured to determine the representative depth value based content in the color map.
The processor maybe further configured to determine a zone range in which a single image is recognized by binocular vergence according to a focal distance and determine the representative depth value based on the zone range.
The variable focus optical system may comprise at least one lens adjusted along an optical axis based on the representative depth value such that a distance between the at least one lens and the display device is adjusted.
The variable focus optical system may comprise at least one lens having a variable curvature.
The transfer optical system maybe configured to transfer the 3D image focused on the reference plane to the pupil of the observer together with a real environment image in front of the observer.
The transfer optical system may comprise a path converting member configured to deflect a path of the 3D image focused on the reference plane in a direction toward the pupil of the observer.
The path converting member may comprise a material configured to transmit a real environment image in front of the observer.
The path converting member may comprise a half mirror.
The 3D image display device maybe a wearable see-through type display device.
According to another aspect of the disclosure, there is provided a three-dimensional (3D) image display method comprising: modulating light according to 3D image information to form a 3D image; focusing, by a variable focus optical system, the 3D image formed on a reference plane; determining a representative depth value of the 3D image by selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, as the representative depth value; adjusting the reference plane by adjusting a focal point of the variable focus optical system based on the representative depth value; and transfer the 3D image focused on the reference plane to a pupil of an observer.
The selecting of the representative depth value may comprise determining the representative depth value based on a depth map and a color map of the 3D image.
The determining of the representative depth value may further comprise: quantizing depth values included in the depth map into a plurality of quantized values; and selecting at least one of the plurality of values as the representative depth value.
The determining of the representative depth value may further comprise determining the representative depth value based on saliency information from the color map.
The determining of the representative depth value may further comprise: determining the representative depth value based on content in the color map.
The determining of the representative depth value may further comprise: determining a zone range in which a single image is recognized by binocular vergence according to a focal distance; and determining the representative depth value based on the zone range.
The adjusting of the variable focus optical system may comprise adjusting positions of at least one lens in an optical axis or adjusting curvatures of the at least one lens.
According to another aspect of the disclosure, there is provided a three-dimensional (3D) image display device comprising: a processor configured to: receive image information to form a 3D image, the image information including a color map of the 3D image; determine a representative depth value of the 3D image based on the color map of the 3D image, and generate a signal to control a variable focus optical system to adjust a reference plane based on the representative depth value.
The processor maybe further configured to determine the representative depth value by selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, as the representative depth value based on visual concentration information from the color map.
The processor maybe further configured to: determine a first zone in the color map of the image having a higher visual concentration than a second zone in the color map of the image as a representative zone, and determine the representative depth value by selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, corresponding to the representative zone as the representative depth value.
The processor maybe further configured to: determine a plurality of zones of comfort (ZOC) based on a relationship between vergence and focal distance corresponding to the 3D image; wherein the determining the representative depth value comprises determining a first representative depth value for a first ZOC and a second representative depth value for a second ZOC.
According to another aspect of the disclosure there is provided a three-dimensional (3D) image display method comprising: receiving image information to form a 3D image, the image information including a color map of the 3D image; determining a representative depth value of the 3D image based on the color map of the 3D image, and generating a signal to control a variable focus optical system to adjust a reference plane based on the representative depth value.
The determining the representative depth value may comprise selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, as the representative depth value based on visual concentration information from the color map.
The 3D image display method may further comprise determining a first zone in the color map of the image having a higher visual concentration than a second zone in the color map of the image as a representative zone, and determining the representative depth value by selecting a depth position, from among a plurality of depth positions corresponding to the 3D image, corresponding to the representative zone as the representative depth value.
The 3D image display method may further comprise determining a plurality of zones of comfort (ZOC) based on a relationship between vergence and focal distance corresponding to the 3D image; wherein the determining the representative depth value comprises determining a first representative depth value for a first ZOC and a second representative depth value for a second ZOC.
The above and/or other aspects will become apparent and more readily appreciated from the following description of example embodiments, taken in conjunction with the accompanying drawings in which:
Example embodiments will now be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present example embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the example embodiments are described below, by referring to the figures, to explain aspects. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
In the drawings, the sizes of elements may be exaggerated for clarity of illustration. The example embodiments described herein are for illustrative purposes only, and various modifications may be made therein.
In the following description, when an element is referred to as being “above” or “on” another element, it may be directly on the other element while making contact with the other element or may be above the other element without making contact with the other element.
As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising” used herein specify the presence of stated features or elements, but do not preclude the presence or addition of one or more other features or elements.
An element referred to with the definite article or a demonstrative pronoun may be construed as the element or the elements even though it has a singular form.
Operations of a method may be performed in appropriate order unless explicitly described in terms of order or described to the contrary. That is, operations are not limited to the order in which the operations are described. In the present disclosure, examples or exemplary terms (for example, “such as” and “etc.”) are used for the purpose of description and are not intended to limit the scope of the inventive concept unless defined by the claims.
The 3D image display device 100 may include an image forming optical system 120 that includes a display device 121 that modulates light to form an image, the variable focus optical system 130 that focuses an image formed by the display device 121 on a reference plane, a processor 150 that controls a focal position of the variable focus optical system 130 such that the reference plane is variable, and a transfer optical system 140 that transfers the image focused on the reference plane to an observer's pupil. According to an example embodiment, the reference plane may be predetermined.
The display device 121 may modulate light according to 3D image information to be displayed to an observer to form an image. The image formed by the display device 121 may be, for example, a stereo image provided to each of the left and right eyes of the observer. In the drawings, only an optical system facing a single eye is shown for convenience of illustration. The image formed by the display device 121 is not limited thereto and may be a hologram image, a light field image, an integral photography (IP) image, or the like, and may also include a multi-view or super multi-view image.
The display device 121 may include, for example, a liquid crystal on silicon (LCoS) device, a liquid crystal display (LCD) device, an organic light emitting diode (OLED) display device, a digital micromirror device (DMD), and may also include a next generation device such as a micro LED, a quantum dot (QD) LED, etc.
The image forming optical system 120 may include a light source providing light for forming an image on the display device 121. In addition, the image forming optical system 120 may further include a beam splitter for adjusting an optical path, a relay lens for zooming in and out an image, a spatial filter for removing noise, and the like.
The 3D image display device 100 according to an example embodiment may include the variable focus optical system 130 for reducing visual fatigue as a configuration for a multi-depth representation. For example, even in the case of a stereoscopy method of recognizing a 3D image by using parallax of the left and right eyes, a position of the reference plane on which left and right eye images are focused may be varied according to depth information included in the image, and thus, visual fatigue may be reduced.
The variable focus optical system 130 may display an image on a plurality of depth positions according to depth information included in a 3D image to be displayed. The variable focus optical system 130 may have a configuration that the focal position varies such that the image formed by the display device 121 is focused on the reference plane and the position of the reference plane is adjusted. The variable focus optical system 130 may vary the position of the reference plane within a depth range DR. According to an example embodiment, the depth range DR may be predetermined. As shown, a plurality of layers LA within the depth range DR may include images corresponding to depth information on reference planes at different positions. The 3D image display device 100 according to an example embodiment may focus an image only on a reference plane of a position D2 selected as a representative depth RD among depth positions D1 to D6 of the reference plane on which the plurality of layers LA are placed. One representative depth position may be selected with respect to an image of one frame as shown. Alternatively, a plurality of representative depth positions may be selected according to images. For example, when depth representation is difficult with only one representative depth, a plurality of representative depth positions may be selected. Even in this case, the number of representative depth positions is selected as the minimum number as will be described later.
Unlike the example embodiment, when all layers related to depth information included in one image frame are displayed, the configuration and operation of the system become complicated and the data throughput increases. For example, when a plurality of layers are displayed using one display device, the plurality of layers may be time sequentially displayed in synchronism with the focus change of the variable focus optical system 130, that is, by a time-division method. In this case, high-speed variable focus devices and display devices in proportional to the number of layers are required. Alternatively, a space-division method may be used. In this case, a plurality of display devices corresponding to the number of depth positions need to be provided, which may increase the size of the system.
The 3D image display device 100 according to an example embodiment may drive the variable focus optical system 130 to select the representative depth and form the reference plane at the corresponding position. Accordingly, a 3D image with reduced visual fatigue may be implemented with a simple configuration. Also, since the representative depth for representing multi-depths is determined through image analysis, an eye tracking sensor for tracking a focus position viewed by the observer is unnecessary and thus the system configuration may be simplified.
The variable focus optical system 130 may include one or more lenses. The one or more lenses may be configured to vary a curvature or move along an optical axis OP, thereby varying a position on which an incident image is focused. Although only one lens is shown in the variable focus optical system 130, this is for convenience, and one or more lenses and a motor or an actuator for providing driving force to the lenses may be provided.
The variable focus optical system 130 may include a curvature variable lens 131 in which a curvature of a lens surface is adjusted. The curvature variable lens 131 may be a liquid lens in which a shape of the lens surface is controlled under electrical control. The curvature variable lens 131 may include a translucent substrate S and an optical membrane M forming the lens surface. An optical fluid F may be filled in a space between the translucent substrate S and the optical membrane M. The curvature variable lens 131 may be controlled by a controller (not shown) that controls the flow of the optical fluid F. According to an example embodiment, the curvature variable lens 131 may include the controller. The optical membrane M may be formed of a transparent and elastic material such that the shape of the lens surface formed by the optical membrane M changes according to the flow of the optical fluid F, and thus, a focal length changes. For example, when the optical membrane M has a first lens surface LS1, a focal plane may be formed at a position FP1, and when the optical membrane M has a second lens surface LS2, the focal plane may be formed at a position FP2. The position of the focal plane may be set according to a selected representative depth and may be adjusted by controlling the flow of the optical fluid F such that a proper lens surface is formed.
The variable focus optical system 130 may include a driving lens 132 that moves in an optical axis direction. The driving lens 132 may be controlled by a driver (not shown) to move in parallel in the optical axis direction. According to a position of the driving lens 132, for example, a focal plane may be formed at the position FP1 or FP2 as shown. The position of the focal plane may be set according to a selected representative depth and may be adjusted by controlling a movement distance of the driving lens 132 to move to a proper position.
Referring again to
The path converting member 145 may include a half mirror. A part of the light L1 including the image formed by the display device 121 may be reflected on a half mirror surface 145a and directed to the observer's pupil and a part of the light L2 including the RE image may be transmitted and directed to the observer's pupil.
In addition to the path converting member 145, the transfer optical system 140 may further include another optical member. Although one lens 142 is shown in the figure, this is an example, and additional lenses or a light path control member may be further provided.
The transfer optical system 140 may transfer the light L1 including the image formed by the display device 121 to the observer's pupil together with the light L2 including the RE image, thereby enabling the 3D image display device 100 to function as a see-through type display. The transfer optical system 140 is shown as including one lens 142 and one path converting member 145, but this is an example and is not limited thereto. For example, in addition to the illustrated half mirror, the path converting member 145 may employ an optical window having various shapes and structures capable of transferring the image formed by the display device 121 to the observer's pupil together with the RE image.
The processor 150 may select a representative depth of a 3D image to be output to the observer and control the variable focus optical system 130 such that the focus of the variable focus optical system 130 is formed at a position of the selected representative depth.
The processor 150 may extract the representative depth by analyzing a depth map and a color map with respect to the 3D image. According to an example embodiment, the processor 150 may perform image content analysis and saliency information analysis to extract the representative depth. Also, the processor 150 may quantize depth values included in the depth map into a plurality of predetermined values and may select one or more of the depth values as the representative depth. Also, the processor 150 may select the representative depth through a zone of comfort (ZOC) analysis considering a human visual perception characteristic. One representative depth or a plurality of representative depths may be selected. When the plurality of representative depths are selected, the minimum number of representative depths as possible may be selected considering the ZOC.
The processor 150 may be configured to control the entirety of the 3D image display device 100. For example, the processor 150 may control a light source (not shown) included in the image forming optical system 120 and a modulation signal for modulating light from the light source in the display device 121.
According to the 3D image display method according to an example embodiment shown in
In the 3D image display method according to an example embodiment, the selection of the representative depth for representing multi-depths may be performed according to image analysis. Therefore, the system configuration may be simplified since an eye tracking sensor for tracking a focus position viewed by an observer is unnecessary.
Referring to
To select the representative depth value, the depth map and the color map may be extracted with respect to the 3D image to be displayed (S210). The color map may be expressed as a color image as shown in
Next, content analysis and/or saliency information analysis may be performed on the color map (S220). The saliency information analysis may be performed to select a zone that is likely to be viewed by an observer, that is, a zone with high visual concentration. Brightness, color, contour, object size, etc. may be considered for the selection of the zone with high visual concentration. For example, a zone having a large brightness or color difference compared to surroundings, a zone having a strong contour characteristic, or a zone having a large object size may be the zone with high visual concentration. A depth value corresponding to such a zone may be selected as the representative depth value. Alternatively, a depth value of a position with high visual concentration may be selected as the representative depth value, according to content in the image.
For instance, according to an example embodiment, a first zone in the color map of the image having a higher visual concentration than a second zone in the color map of the image is determined as a representative zone, whose position is selected as the representative depth value. The visual concentration information may be one or more of brightness, color, contour, object size, etc. According to an example embodiment, the visual concentration information may be a characteristic of the image that draws the viewers concentration or attention to a particular zone or region in the image.
In addition to the selection of the representative depth value, human visual perception characteristics may be considered. To this end, a ZOC analysis may be performed (S250) and the minimum number of representative depth values may be selected considering the ZOC analysis.
Vergence means concentration of visual axes of both eyes on an object to be viewed. When a 3D image is perceived by binocular parallax, vergence is a phenomenon that both eyes converge toward the object to be viewed. Vergence occurs to perceive a single image when binocular parallax exceeds a parallax fusion limit and a double image is experienced. Convergence occurs when the eyes move from a far place to a near place. To the contrary, divergence occurs when the eyes move from a near place to a far place. Accommodation occurs in synchronization with such an action. When the focal distance and a vergence distance coincide with each other, a clear image is recognized. In the human visual perception characteristic, an image of a predetermined range in which the focal distance and the vergence distance do not completely match may be recognized as the clear image. In the graph in
Referring to the graph in
For example, considering the ZOC, a depth range of about 30 to about 100 cm may be set as a zone Z1 that covers a predetermined vergence position range, and a depth range of about 100 to about 500 cm may be set as a zone Z2 that covers another predetermined vergence position range. In this case, a depth expression of about 30 to about 500 cm may be expressed as two representative depth values RD1 and RD2 which are respectively selected in the two zones Z1 and Z2.
The representative depth value RD1 may be selected as an arbitrary value in the depth range of about 30 to about 100 cm. The representative depth value RD2 may be selected as an arbitrary value in the depth range of about 100 to about 500. In
With regard to selection of the representative depth values, the depth values may be quantized (S260) and the representative depth value may be selected from the quantized values (S270).
Depth information included in a 3D image may be represented as, for example, 256 depth values of 8 bits. Quantization of depth values means that not all of the depth values are used for the depth representation, but some discontinuous values of the depth values are selected for the depth representation. For example, 8 depth values, d0, d32, . . . , d232, may be selected from depth values d0 to d255. However, the 8 depth values here are examples and any value smaller than 256 may be selected. The quantization of depth values described above is intended to reduce the fluctuation of representative depth values selected in successive frames when, for example, a representative depth value is selected for every frame for display of moving pictures. Even when a representative depth value is selected from the depth values before quantization and there is a difference in the representative depth values of successive frames, the same representative depth value may be selected in successive frames when the representative depth value is selected from the quantized values. When the same representative depth value is selected in successive frames, since driving of the variable focus optical system 130 is not required, driving of the 3D image display device 100 may be further simplified.
The representative depth value may be selected considering a ZOC analysis and the quantization of depth values together. For example, in
As described above, one representative depth value or the minimum number of representative depth values as possible when there are one or more representative depth values may be selected by reflecting saliency information analysis and/or content analysis together with the ZOC analysis and the quantization of depth values to an image of one frame.
When a representative depth value is selected, a variation value of the variable focus optical system 130 may be set according to the representative depth value (S310). The variation value may be the curvature of a lens or a movement distance of the lens according to the configuration of the variable focus optical system 130.
Next, a control signal capable of implementing the variation value (the curvature or the movement distance) may be set (S330). The control signal may be, for example, an electric signal for forming a hydraulic pressure for varying a curvature of the curvature variable lens 131 of
According to the set control signal, the variable focus optical system 130 may be driven to form a focal plane at a predetermined position (S350).
Referring again to
The 3D image display device described above may be configured in a wearable form. All or some of components of 3D image display devices may be configured in the wearable form.
For example, the 3D image display device may be applied in the form of a head mounted display (HMD). Further, the present disclosure is not limited thereto. The 3D image display device may be applied to a glasses-type display or a goggle-type display.
The 3D image display device described above may be applied to implement AR display in that an image formed in an image forming optical system and an image of the real world are displayed together to an observer.
AR display may further increase the effect of reality by combining and displaying virtual objects or information on an environment of the real world. For example, at a position of the observer, additional information about the environment provided by the real world may be formed by an image former and provided to the observer. The AR display may be applied to a ubiquitous environment or an internet of things (IoT) environment.
The image of the real world is not limited to a real environment and may be, for example, an image formed by another image device. Accordingly, the 3D image display device described above may be applied as a multi-image display device that displays two images together.
The 3D image display device described above may be operatively connected to or linked to another electronic device such as a smart phone. For example, a processor driving a 3D image display device may be provided in the smart phone. In addition, the 3D image display device described above may be included in the smart phone such that the smart phone itself may be used as the 3D image display device.
According to the 3D image display device and method described above, multi-depth may be represented and thus a 3D image without visual fatigue may be formed.
According to the 3D image display device and method described above, the minimum number of selected representative depths may be used for each image frame with regard to representation of the multi-depths, the system driving may be simplified, and the data throughput may be reduced.
According to the 3D image display device and method described above, since the representative depth for representing multi-depths may be selected by image analysis, a separate eye tracking sensor is unnecessary, and thus the system configuration may be simplified.
The 3D image display device and method described above may be easily applied to a wearable device, and may be applied to, for example, a glasses-type AR display device.
Example embodiments have been described and illustrated in the accompanying drawings to help understanding of the disclosure. However, these embodiments are merely examples not limiting the scope of the disclosure. Also, it will be understood that the disclosure is not limited to those illustrated and described in the disclosure. That is, those of ordinary skill in the art may make various modifications therein.
It should be understood that example embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example embodiment should typically be considered as available for other similar features or aspects in other embodiments.
While one or more example embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2018-0088674 | Jul 2018 | KR | national |
