Patent Application
20250199350
This US patent application claims the benefit of EP patent application No. 23465572.8, filed Dec. 13, 2023, which is hereby incorporated by reference.
The disclosure relates to a display with integrated camera and a vehicle with such integrated display.
Given the current trends in display technology, more and more OEMs (OEM: Original Equipment Manufacturer) from different sectors are looking to integrate a camera behind a display within their products. This type of approach can be found in consumer products such as smartphones, tablets, and laptops, currently raising the interest for automotive products also.
In addition to this, driver monitoring systems are becoming mandatory in order to be able to satisfy the new safety regulations, therefore the camera systems are starting to be integrated in different devices across a vehicle, such as cars, trucks, buses and so on.
In order to address this matter, various approaches have been used by OEMs and manufacturers from different industries looking to achieve seamless camera integration in their devices.
The technology used to integrate the camera behind a display is mainly influenced by the display type technology, such as LCD (LCD: Liquid Cristal Display) display and LED (LED: Light-Emitting Diode) display.
Smartphone manufacturers addressed this problem by firstly creating a notch within the display and installing the camera within the notch behind a cover glass of the display. However, such camera installed in the notch cannot be regarded as seamless integrated and as a camera behind a display.
To achieve a seamless camera integration behind a display, multiple options were identified. For example, EP 4 102 810 A1 and U.S. Pat. No. 11,145,233 B1 describe the usage of a lower pixel density and/or of smaller pixels, for an active area of the camera. Hence, such approaches propose a lower pixel density and/or smaller pixels for the area where the camera captures the image to create enough space for the light to enter. Different pixel arrangements are also possible for such approaches, as this aspect can also have an effect for the quality of the picture. However, such solutions that are using lower pixel density and/or smaller pixels for the active area of a camera would result in that each platform or application would need a specially designed LCD panel to meet the requirements of the product, which additionally would result in an increase of complexity of the design and an increase in the costs of such solutions.
US 2020/0117034 A1 describes a camera installed behind an LCD display in a through hole. The LCD display comprises a cover glass, an optically clear adhesive (OCA: optically clear adhesive) layer, a first polarizer, a first glass substrate, a color film (CF: color film) layer, a thin film transistor (TFT: thin film transistor) layer, a second glass substrate, a second polarizer, and a backlight module. The through hole is provided through the first polarizer, the CF layer, the TFT layer, the second polarizer, and the backlight module. The through hole forms a light channel in the LCD for ambient light coming from outside of the electronic device, passing through the cover glass and the light channel, and entering into the camera. The camera is disposed in whole or in part inside the through hole. The first glass substrate and the second glass substrate are not hollowed out in a region corresponding to the light channel. However, such an approach requires a great deal of effort to align such a through hole in almost all parts of an LCD. Furthermore, drilling such through hole in nearly all parts of the LCD can result in damages in each single part of the LCD. Hence, the parts must be controlled for damages and a correct alignment with regard to the through hole. All the afore mentioned in turn leads to higher costs.
WO 2021/258921 A1 describes a camera device under a display screen. The display comprises a display screen and a camera. The display screen includes a cover at the top and a display module under the cover. The display module includes an ITO (ITO: Indium Tin Oxide) glass, a liquid crystal layer and a backlight layer. The display screen is provided with a blind hole area, which blind hole area is formed on the liquid crystal layer above an opening that goes through the whole backlight layer. The camera is arranged directly below the blind hole and a lens of the camera extends into the hole in the backlight layer onto the blind hole area, forming an angle of view through the blind hole. The display screen further includes an electric field circuit, which is provided on the ITO glass layer of the outermost layer of the liquid crystal layer, and which electric field circuit drives the deflection of the liquid crystal in the blind hole area of the display screen through the change of the electric field, so as to change the optical path of the light passing through the blind hole area, and to realize the focusing of the camera. Such approach is complicated to realize and means higher costs. Furthermore, a complete hole through the whole backlight layer would influence a light guide, in that the scatter of the light being disturbed in the area of the hole, resulting in image capture issues.
Furthermore, the development of seamless camera integrated devices is challenging due to the opacity of LCD panels and polarizers. LCD panels are composed of a liquid crystal layer sandwiched between two electrodes, together with the polarizers that are optical filters that transmit light of a particular polarization. Usually, common LCD panels have two polarizers, one for each side. When light passes through an LCD panel and polarizer, it is partially absorbed and scattered, which reduces the amount of light that reaches a camera sensor behind a display. This can lead to decreased image quality and reduced camera performance, thus an increased amount of hardware compensation is needed.
US 2022/0179452 A1 describes electronic devices including a front-facing camera disposed behind a front-facing display. Such device described in US 2022/0179452 A1 comprises a display having a front surface and a back surface, the display including a plurality of pixel regions that emit light from the front surface of the display to display a displayed image and a plurality of apertures that transmit light from the front surface to the back surface. A camera is disposed on a side of the back surface of the display. The camera is configured to capture images. The device further includes restoration optics disposed between the display and the camera, which restoration optics are configured to reduce image distortion caused by the display. It is further described that the restoration optics may include an optical filter (e.g., a phase-shift mask), a bandpass filter, a lens that includes a varifocal lens and/or different and/or additional components. It is further described that various items of the restoration optics can be included in the camera and therefore excluded from the restoration optics. Furthermore, a processor is coupled to the display and the camera, which processor is configured to apply a digital filter to at least a portion of the captured image to further reduce the image distortion caused by the display. Accordingly, the image distortion caused by the display is reduced both physically, by the restoration optics and digitally by the processor. Hence, a complex setup is described in US 2022/0179452 A1 that needs both physically and digitally reduction of image distortion caused by the display. However, such approach also leads to high efforts and results in high costs.
WO 2009/051673 A1 describes an image capture device, working in the visible spectrum, being placed behind a switchable diffuser and either behind a light guide or in a hole or opening within the light guide behind a display. The switchable diffuser is arranged in front of the light guide. WO 2009/051673 A1 further describes that a light guide typically has scattering elements, also referred to as scattering dots, on the backside of the light guide plate to scatter light that has been conducted from the backlight bulbs into the light guide plate, toward the front side of the LCD display assembly. Such scattering elements are also known from the state of the art as dot structures, microdots or microlenses, provided at a surface of the light guide, typically the backside of the light guide, used as described in WO 2009/051673 A1 to guide light towards the display panel. The amount of resulting light is defined by the density of a dot structure, microdot or microlens population. The higher the density, the more light will leave the light guide at its desired area. However, smaller densities are limited, as single microdots or microlenses will not produce homogeneous light output. While the dot structures are known as small printed white dots on the backside of the light guide, the microdots or microlenses are imprinted or injection molded microdots of hemisphere-like shape with spherical caps provided on a surface on the backside or a frontside of the light guide plate. The diameter of the spherical caps or rather the microdots or microlenses have a size of around tens of micrometers.
WO 2009/051673 A1 further describes the importance of the usage of the switchable diffuser to be able to capture sufficient light for the image capture device, to improve the uniformity of the light going through the LCD panel and to make the scattering elements less visible from a front side of the LCD.
Furthermore, to be able to obtain a clear image, the switchable diffuser, the LCD and the image capture device have to be accurately controlled to enable the camera to capture an image. WO 2009/051673 A1 therefore proposes a control logic for the LCD panel, camera and diffuser, to permit sufficient light to be captured by the camera. Hence, the solution known from WO 2009/051673 A1 needs an increased amount of logic and Hardware to perform all the mandatory steps in order to capture the image. Moreover, it can be difficult to integrate this type of solution in lower end, lower cost products considering the control logic needed to switch the diffuser and the LCD screen.
It is therefore an object of the present disclosure to provide an improved display with a seamless integrated camera behind the display.
This object is achieved by a display with integrated camera and a vehicle with such integrated display according to the claims. The claims include further developments and improvements of the present principles as described below.
According to a first aspect, an apparatus comprises a display device and a camera. The display device comprises a cover glass, an LCD Panel, and a backlight unit. The LCD Panel is arranged behind the cover glass, the backlight unit is arranged behind the LCD Panel and the camera is arranged behind the backlight unit with its field of view directed to the backlight unit, more precisely to a light guide of the backlight unit further described below. The cover glass is the part of the display that is closest to a user looking at the display. For example, behind the cover glass means that the respective component behind the cover glass, in this case the LCD panel, is arranged on an opposite side of the display glass than the side of the display glass that a user is looking at when looking at the display. The field of view of the camera defines an active area of the camera. It is understood that the field of view and therefore the active area of the camera is larger if, for example, a wide-angle lens is used and/or the further away the camera is placed from the backlight unit. It is also understood that, conversely, the field of view and therefore the active area of the camera is smaller if, for example, a telephoto lens is used and/or the closer the camera is moved to the backlight unit.
The backlight unit comprises an edge light source, a light enhancement layer and a light guide. The light guide is arranged behind the light enhancement layer. The camera operates in the infrared wavelength spectrum. The light enhancement layer may comprise one or more foils. The advantage of the usage of a camera which operates in the infrared wavelength spectrum is that the IR light transmission is not or only weakly affected by the LCD panel and its polarizers. The term “operates in the infrared wavelength spectrum” means that the camera or rather an image capture element of the camera is at least sensitive to the infrared spectrum and at least captures images in the IR-spectrum (IR: infrared). It is understood that the camera may either be only operatable in the IR-spectrum, i.e., be arranged as an IR-Camera, or in addition to operating in the IR-spectrum also operating in the visible wavelength spectrum. The light enhancement layer comprises a cut-out window in the active area of the camera. It is understood that the cut-out window, preferably is identical to the size of the active area. However, the cut-out window may be smaller than the size of the active area. The cut-out window has less disturbing visible effect to the user watching at the display, e.g, a driver. Alternatively, the cut-out window may be larger than the size of the active area. No negative influence of the light enhancement layer appears in the camera image. The term “cut-out window” in the light enhancement layer means that there is an area in the light enhancement layer defined by the active area of the camera, where the light enhancement layer is modified, for example there is a hole in a part of the light enhancement layer, and through which modified area the camera may look through in an undisturbed manner.
Furthermore, the light guide is a light guide with nanostructures instead of microstructures. It has been found that a light guide with nanostructures does not affect infrared light when compared to a light guide with microstructures. Hence, a light guide with nanostructures is suitable to be used with cameras operating in the IR wavelength spectrum, with no further processing needed. A surface of the light guide may be provided with such nanostructures at a backside or a frontside of the light guide. However, when applied to the backside surface, such structures become less visible. The term “nanostructures” mean that the structures, e.g., spherical caps at a surface of the light guide, have sizes smaller than 1 μm with regard to their diameter or rather its largest extent in one dimension. Preferably, the structures have sizes between 1 nm and 1 μm. Particularly preferred, the structures have sizes between 100 nm and 990 nm. The form, size and density of the elements, i.e., the density per a given area, can be individually chosen depending on the used light guide and the technical requirements of the whole apparatus, e.g., the display. For example, the form of such nanostructure can be similar to a pyramid, a cone, a frustum and/or a cylinder instead of spherical caps. It is understood that the invention is not limited to the afore-mentioned forms of structures and that there exist plenty further forms that may be used for those structures.
The present disclosure proposes a simple solution to obtain a seamless integration of a camera, e.g., a driver monitoring module, in a display, with no effect on the displayed content. E.g., the present solution removes the need for a switchable diffuser and increased logic to control it, given that the camera may capture an image only when the diffuser is set to “clear” state and the LCD panel to “transparent” state. The solution further removes the need of a through hole through parts of the LCD panel, the whole backlight unit and or the need to drive the deflection of the liquid crystal in a blind hole area of the display screen through the change of an applied electric field, to change the optical path of the light passing through the blind hole area, and to realize the focusing of the camera. Furthermore, with the solution there is no necessity of a physical reduction of an image distortion caused by the display, e.g., by using restoration optics.
Moreover, the proposed solution can accommodate the camera in any location of the LCD panel, given the fact that the solution may use any commercially available LCD screen with no additional processing needed. The proposed seamless integration of the camera behind the display provides an enhancement in user experience, yet with additional functionalities for the device.
According to an embodiment of the present application, the cover glass is optically bonded to the LCD Panel. The optical bonding is clear and it is understood that the cover glass and optical bonding material are transparent and do not affect the behavior of a captured image. By optically bonding the cover glass to the LCD Panel, an air gap between the cover glass and the LCD panel is eliminated and thus, e.g., the display performance, readability, touch accuracy, and impact resistance is improved.
According to another embodiment of the present disclosure, the light enhancement layer is formed by a stack of foils comprising a Louver foil and/or a reflective polarizing foil and/or a diffusing foil at least one of which comprises the cut-out window. It is understood that dependent on the respective foil's properties, it may be sufficient if only one of the foils of the stack of foils comprises a cut-out window. For example, several foils of such stack of foils may not have or have only little effect on IR-light. The usage of a foil stack with foils having different properties has the advantage that the optical performance of such apparatus may be enhanced dependent on the technical requirements of the apparatus, e.g., the display.
According to another embodiment of the present application, the apparatus comprises a processor, the processor is provided to apply a post-processing algorithm, preferably a deconvolution algorithm, on a captured image taken by the camera. The post-processing algorithm enhances the quality of a captured image with regard to effects caused by the LCD panel on a captured image. It is understood that the post-processing algorithm preferably is a deconvolution algorithm. However, there exist further post-capture image processing methods to improve image quality of a captured image which may also advantageously used as such post-processing algorithm.
According to another embodiment of the present application, the size of the cut-out window is smaller than the active area. The camera is detecting image information of a first area within the cut-out window and of a second area outside the cut-out window. The detected image information from both areas is provided to the post-processing algorithm. Such embodiment provides information to calibrate the post-processing algorithm.
According to another embodiment of the present application, the LCD Panel comprises two polarizers at least one of which does not influence or only weakly influence IR light passing the respective polarizer. Such embodiment improves light yield in the IR-spectrum with the camera behind the display. It is understood that such embodiment may comprise a special physical arrangement of the at least one polarizer which enables at least a part of IR-light to pass the at least one polarizer. Preferably, a part of the at least one polarizer in the line of sight of the camera or the whole at least one polarizer comprises or consists of a material which enables at least a part of IR-light to pass the at least one polarizer.
According to another embodiment of the present application, the LCD Panel comprises two polarizers, both of the polarizers do not influence or only weakly influence IR light passing the polarizers. Such embodiment further improves light yield in the IR-spectrum with the camera behind the display. It is understood that such embodiment may comprise a special physical arrangement of both polarizers which enables at least a part of IR-light to pass both polarizers. Preferably, a part of each of both polarizers in the line of sight of the camera or both polarizers in a whole comprise or consist of a material which enables at least a part of IR-light to pass both polarizers.
Preferably, the liquid crystal material of the LCD panel does not influence or only weakly influence IR light passing the LCD Panel. Such embodiment further improves light yield in the IR-spectrum with the camera behind the display, similar as described above with regard to polarizers.
According to a second aspect a vehicle comprises an apparatus in accordance with at least one of the afore-mentioned embodiments.
Yet described with regard to a usage of such display with seamless integrated camera behind the display, furthermore, such seamless integrated camera behind a display may be introduced in products that are following a user and provide an interactive experience given the orientation of the user's eyes, especially for entertainment activities. For example, the camera may be introduced in the back of a Smart TV display, and through specific and compatible content, the user may have a real-time experience with his digital twin like playing different games or participating in sport classes. A further example is an apparatus with a seamless integrated camera behind a display that additionally comprises a motion sensor input device that enables features such as skeleton tracking, hand interactions, and voice recognition. Such motion sensor may track the motion of the person in front of the device and create a link between the movement and a “digital twin” shown on the display. In order to do this, such motion sensor input device is equipped with RGB cameras and infrared projectors and detectors that may perform real-time gesture recognition. By placing the camera solution according to the disclosure behind the display of the monitor or TV a better experience for the user may be obtained, providing an integrated, seamless solution without the need of a second device.
Further features of the present disclosure will become apparent from the following description and the appended claims in conjunction with the figures, wherein:
For a better understanding of the principles of the present disclosure, embodiments of the disclosure will be explained in more detail below with reference to the figures. Like reference numerals are used in the figures for the same or equivalent elements and are not necessarily described again for each figure. It is to be understood that the disclosure is not limited to the illustrated embodiments and that the features described may also be combined or modified without departing from the scope of the disclosure as defined in the appended claims.
The difference in the embodiment shown in
The light guide 5 comprises nanostructures 11 which may be seen in more detail in an exemplary symbolic form in an enlarged view of the active area 9 of the camera 3, shown in
In
| Number | Date | Country | Kind |
|---|---|---|---|
| 23465572.8 | Dec 2023 | EP | regional |
