Patent Application
20230236546
The present invention relates to methods and apparatus for controlling the illumination power of a Computer-Generated Hologram (CGH) display.
A computer-generated holographic image is produced when at least partially coherent light is incident on a display device capable of modulating the wavefront of incident light in such a way as to create an interference pattern that sums to yield an image to an observer looking directly or indirectly into the display. The pattern displayed on the device is referred to as the hologram. The hologram, through modulating the amplitude, phase or a combination of both, steers the incident light into a replay field to create an image.
A characteristic of such a CGH image is that the total power of light is conserved in the output. As a result, objects that appear large in the replay field of the hologram will appear dimmer than objects that appear smaller, for the same power of light incident on the display. Therefore, for the same illumination power, holographic images of varying sizes will have different brightness. This can reduce image quality, reduce the overall viewing experience, lead to incorrect reproduction of colours and/or potentially create a dangerously high brightness (such as if the entire laser power is focused on a single spot).
It would be desirable for a CGH to have more consistent apparent brightness between images of different sizes in the replay field.
Embodiments of the present invention make use of an energy or intensity calculation at a frame or sub-frame level to adjust an optical power of an illumination source of a computer-generated hologram display such that the apparent brightness of a replay image is consistent with the brightness of an original scene used for the CGH. Other embodiments provide ways for such energy or intensity to be communicated between elements of a holographic display system in a way such as to maintain synchronisation between the illumination power and the displayed CGH.
While backlight power control has been proposed for some display technologies, such as Liquid Crystal Displays (LCDs), this focuses on expanding an apparent dynamic range of the display; expanding the contrast ratio or the perceived difference between light and dark sections of an image. An LCD operates by blocking the backlight to some degree so that illumination from the backlight is not conserved in the output. When backlight dimming is used with an LCD, the light required to be blocked for dark pixel is reduced, so dark colours appear darker. However, the total power of illumination is not conserved in the way it is with CGH displays. Unlike backlight dimming for contrast enhancement, the embodiments herein control energy for more consistent brightness across different displayed CGH images. In addition, when images are displayed on non-holographic displays, such as LCDs, the image data sent to the display includes scene energy (brightness) information, so that the frame data itself can be used to determine a required brightness. In contrast, a holographic replay field does not inherently include scene energy information; this is lost in the conversion to the hologram, so it is advantageous to determine a scene energy from the source data.
According to a first aspect of the present invention, there is provided a method for adjusting the apparent brightness of a computer-generated hologram display. The method comprises: receiving source data representative of a scene to be displayed as a hologram; determining hologram data to display a computer-generated hologram representing the scene; determining a scene energy based on the source data; associating the scene energy with the hologram data; and controlling a holographic display according to the hologram data and simultaneously controlling an output power of an illumination source of the holographic display according to the scene energy.
Source data may be any suitable data for generating a hologram for display, such as a 3-dimensional array including intensity and/or colour data with an associated depth or position in a reference space, or a collection of points in 3-dimensional space with associated intensity and/or colour data (such as point-cloud data). For example, the source data may be output from a graphics engine, such as the Unity® 3-dimensional engine. Determining hologram data from source data is known to the person skilled in the art. However, when such hologram data is displayed, the apparent brightness of the scene will depend on the occupancy of the replay field. If only a small part of the replay field is occupied with the scene, the scene will appear brighter than if a large part of the replay field is occupied with the scene, for the same output power of the illumination source. The variation in perceived brightness is addressed by determining and using scene energy in the display of the hologram.
Scene energy may also be referred to as illumination energy or intensity. The scene energy is determined based on the source data and so considers the brightness of the source data to be represented by the hologram. The scene energy can be determined at a frame level or a sub-frame level. A sub-frame may be a single colour for display with others to generate a perceived full colour image, such as a red, green and blue sub-frame. One example to determine scene energy is to sum the colour and/or intensity values of all pixels in the bitmap of source data at the frame or sub-frame level.
By associating the scene energy with the hologram data, it can be maintained in substantial synchrony with the hologram data, allowing the illumination source to be controlled to vary its power as each frame or sub-frame is displayed.
The association between the scene energy and hologram data can be formed in various ways, for example using metadata associated with the hologram data, or by providing the scene energy in a header section of the hologram data.
The scene energy may have a very large range of values. The range will scale with the number of pixels, increasing the more pixels there are. In some examples, the scene energy is normalised to a scale between a predetermined minimum apparent brightness and a predetermined maximum apparent brightness. For example, the scene energy may be normalised between 0 and 1, and the scene energy normalised by dividing by the scene energy value for a completely white image. Other factors can also be considered in the normalisation, including one or more of a maximum scene energy for safe viewing of the hologram, a maximum possible output power or intensity of the illumination source, and a minimum energy detectable by human eye in specified conditions.
The many different values for scene energy may mean that it is technically challenging to construct a display system that is both eye-safe (the output power does not exceed that permitted for a Class 1 device under BS EN 60825-1:2014) and able to produce a visually distinguishable change for each small change in scene energy. A holographic display with acceptable quality may be achieved by quantising the scene energy to a smaller range of discrete values falling within the range of visually distinguishable illumination powers that the display equipment can produce. For example, the scene energy might be quantised to an 8-bit or 16-bit number. The scene energy may therefore be quantised on a predetermined scale to reduce the number of different brightness values. While some examples may use a linear scale, other examples may quantise the scene energy on a non-linear scale. A non-linear scale can provide smaller differences between values where the human eye is most sensitive to variations in brightness, or can ensure that there are more visually distinguishable brightness levels for the most common scene energy levels, improving perceived image quality while reducing the number of values required to quantise scene energy. For example, a logarithmic quantisation may be used.
In examples using a non-linear scale for quantisation, the non-linear scale may have a closer spacing between values in a mid-section of the scale then it does between values closer to a minimum and a maximum of the scale. For example, the spacing could use a logarithmic spacing but, rather than distributing the points so that they are closest spaced towards the minimum or the maximum, they can be distributed so that they are closest spaced at a mid-point and with spacing increasing towards both the maximum and minimum. The mid-section may be centred on the halfway point between maximum and minimum, or offset from the halfway point.
As discussed above, the scene energy may be a sum across all pixels and colours in the source data. In that case, the sum may comprise a respective weighting for each colour in the source data. The weighting can account for various different factors, such as one or both of different hardware characteristics when reproducing each colour and different perceived brightness for each colour in the human eye.
The scene energy may also be represented by sums across all pixels for each colour in the source data, each colour corresponding to a colour of the illumination source. This can be useful when colour is displayed by time-multiplexing different colours of illumination source, such as red, green and blue illumination sources.
In some examples, the method comprises monitoring an emitted intensity of the illumination source and using the emitted intensity for feedback control of the output power of the illumination source. This can remove the need to calibrate an output brightness to the scene energy: by monitoring the brightness (such as by using a photodiode or photosensor) feedback control can be used to match the emitted brightness to that indicated by the scene energy. There may also be benefits to feedback control to compensate for changes in the output characteristics of an illumination source during its operation. For example, the output intensity may change as the illumination source ages and/or change with the operating temperature of the illumination source.
Some examples may include monitoring an emitted intensity of the illumination source and shutting off the illumination source if the emitted intensity exceeds a predetermined threshold. This may improve the safety of the system. Some examples may combine this shut off with the feedback control discussed above.
The output power of the illumination source can be controlled in any suitable way. In general, the output power depends on one or more input drive parameters. Example input drive parameters include drive current and pulse width. In one example, controlling the output power of the illumination source uses a combination of current modulation and pulse width modulation (PWM). For example, the output power to the illumination source can be varied both changing an absolute value (through the current modulation) and changing an average value (through PWM). PWM may be expressed in terms of pulse width or duty ratio. The current modulation and PWM can be combined in various ways. One example uses current modulation with constant pulse width above a predetermined threshold and PWM with constant current below the predetermined threshold, or vice versa. Other examples may vary both PWM and current simultaneously. Where there is a switch between PWM and current modulation either side of a predetermined threshold, the total number of levels to control the illumination source intensity may be the sum of the number of levels used for current modulation and the number of levels used for PWM. By varying both current and PWM together, a greater number of levels is possible (the product rather than the sum) but determining the current and PWM level to use may be more complicated.
Some examples may receive data indicative of a temperature of the illumination source. That data can then be used in the controlling the output power of the illumination source, so that the output power is based on the data indicative of the temperature of the illumination source and the scene energy. This can improve the accuracy of the output power of the illumination source. The output power of the laser diode may be dependent on temperature as well as input drive parameters, as mentioned above. By receiving data of the current temperature, the data can be used in the control, such as via a predetermined lookup table or previous calibration of the illumination source. Some examples may combine this data indicative of temperature with the feedback control of monitored output intensity discussed above. This may improve response times for the feedback control to achieve the required output intensity. Other examples may use the data indicative of temperature in isolation of such feedback control. This can allow a simpler construction because temperature data may be collected without impinging the optical path.
In examples, controlling the holographic display according to the hologram data and simultaneously controlling the output power of an illumination source is synchronised based on a signal received from a modulator of the holographic display. A modulator of the holographic display may have a different response time to the illumination source, typically the modulator is slower. By synchronising based on a signal received from the modulator, for example a signal indicating that the modulator has formed an image ready for display, perceived image quality can be improved.
According to a second aspect of the invention, there is provided a holographic display system. The system comprises: an at least partially coherent illumination source; a modulator configured to be illuminated by the illumination source; an input for receiving source data; and a controller configured to operate according to the method of the above-described first aspects, with or without the optional features also described.
The illumination source may be substantially coherent in some examples. Example illumination sources include a laser and a Superluminescent Light-Emitting Diode (SLED). Example modulators include a Spatial light modulator (SLM). The input can receive source data in any suitable way including a wired or wireless connection using existing protocols for transmission of display data, network data and/or serial data. Any combination of software and hardware can be used to implement the controller, for example the controller may be an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA) configured to execute the processing in hardware, or a processing system, such as a microprocessor, graphics processing unit, or programmable Digital Signal Processor (DSP) programmed by computer program code to implement the processing.
According to a third aspect of the present invention, there is provided a method of operating a holographic display. The method comprises: receiving hologram data and associated scene energy data; operating a modulator to modulate a substantially coherent illumination source, thereby to generate a light field corresponding to a frame or sub-frame of the hologram data; and controlling an output optical power of the illumination source incident on the modulator according to a scene energy associated with the frame or sub-frame of hologram data. In this way the perceived brightness of the hologram can be controlled according to the scene energy data. The scene energy data may be determined by another element than the display itself, for example sent from a processing system that is providing hologram data to the holographic display. The controlling of the output optical power may be synchronised based on a signal received from the modulator.
According to a fourth aspect of the invention, there is provided a holographic display apparatus. The holographic display comprises: a substantially coherent illumination source; a modulator configured to be illuminated by the illumination source; an input for receiving hologram data and scene energy data associated with the hologram data; and a controller configured to operate the modulator and the illumination source according to the method of the third aspect. While the illumination source and modulator may be the same as in the second aspect, the controller of this aspect is configured to control display of holographic data without also determining that data and the associated scene energy. This can allow the holographic display to be simpler with lower processing requirements, which may allow lower power operation and improved miniaturisation or integration of components. As with the second aspect, the controller can be implemented as any combination of hardware and software.
According to a fifth aspect of the present invention, there is provided a method of generating computer-generated hologram data. The method comprises: receiving source data representative of a scene to be displayed as a hologram; determining hologram data to display a computer-generated hologram representing the scene; determining a scene energy based on the source data; and associating the scene energy with the hologram data. The scene energy is for controlling an output power of an illumination source when displaying the computer-generated hologram data. The method of this aspect may include any of the optional features of the first aspect discussed above. Unlike the first aspect, a holographic display is not directly controlled. This method could be useful where the display is separate from the encoding for display. Examples include a computing system, games console, image source or video source which generates hologram data for sending to a separate holographic display and/or for storage. Storing the hologram data and associated scene energy can be useful for pre-generating or pre-rendering content for later display, such as pre-rendered media for later reproduction, perhaps in a cinema or in a user's home.
According to a sixth aspect of the present invention, there is provided a method of transferring data to a holographic display. The method comprises: associating scene energy data with a frame or sub-frame of hologram data; and transferring the scene energy data simultaneously with the hologram data. The simultaneous transmission keeps the scene energy data synchronised with the hologram data. The scene energy data may be associated using metadata.
According to a seventh aspect, there is provided a processing system configured to implement the method of at least one of the fifth or sixth aspect. For example, the processing system may be a computing system, games console, image source or video source.
According to an eighth aspect, there is provided an illumination source driver apparatus. The apparatus comprises: an input for receiving scene energy data; a pulse width modulator; an adjustable current source; an output coupled to the pulse width modulator and the adjustable current source for outputting a signal to control an emission power of an illumination source; and a controller. The controller is configured to control a pulse width of the pulse width modulator and a current of the current source based on received scene energy data. The driver apparatus can be used to control the output level of a substantially coherent illumination source for a holographic display according to scene energy data.
The illumination source driver may comprise a sensor for sensing an emission power (or output power) of the illumination source; wherein the controller is configured to control the pulse width and current further based on the emission power. This may allow the driver to control the illumination source more accurately, using feedback control. Some examples may combine the sensing of emission power with control based on data indicative of temperature discussed above.
Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only, which is made with reference to the accompanying drawings.
Example holographic displays comprise a phase modulating Liquid Crystal on Silicon (LCoS) display, illuminated by a coherent or at least partially coherent source, such as a laser source. In this case, the display is a reflective type—the laser beam from the laser source is optically expanded to cover the whole surface of the display containing the hologram and the reflected light forms the holographic image.
Unlike non-holographic displays, the light energy incident on a holographic display is conserved, up to inefficiencies in the display. That is, the energy of the light from the display surface that is diffracted to form the holographic image is substantially equal to the energy of light incident on the hologram. For images of different apparent size, this means that if the same illumination power is used for all images, then the same light power will be distributed into different volumes, with the result that larger holographic images appear dimmer than smaller holographic images. This effect is illustrated diagrammatically in
Unlike non-holographic displays, the brightness of the image therefore depends not only on the brightness or intensity of the illumination source but also on the size of the holographic image in the replay field. This impacts the perceived image quality because holographic images of different sizes will appear at different brightness from each other, even if they all had the same brightness in the source data used to generate the holographic image (such as a representation of a scene which is then rendered as a hologram). In some cases, the holographic images may be sufficiently large that their perceived brightness is so low that they become imperceptible against background ambient light, especially for augmented reality display systems. At the other extreme, a particularly small holographic image may result in an image which is bright enough to cause eye safety issues because the light intensity may exceed an eye safe limit.
Controlling Illumination Power for Consistent Brightness with Size of Image
To maintain constant brightness across varying source scenes, systems discussed herein adjust the illumination power according energy in the source data, for example by using the pixel occupancy and the intensity of the points in each given source scene. In this way, consistent brightness can be achieved between holograms of different sizes. This is illustrated in diagrammatic form in
The upper hologram 8 of
The discussion of
Having outlined the general principle of operation, an example method of operating a holographic display system which can adjust the power of an illumination source for more consistent holographic image brightness will be described.
At block 102, source data is received. In this example, the source data is scene data received from Unity (a 3D content engine). “Receiving” data includes the source data being pushed from a content source or pulling the source data by requesting it from the content source.
Next, at block 104, the holographic data is determined. In this example, the Unity® scenes are provided as input into a software program that computes holograms from the scene data using methods known to the person skilled in the art. For example, a suitable algorithm to provide real-time hologram generation is discussed in “Adaptive Aberration Correction for Holographic Projectors”, Kaczorowski, A. available from https://www.repository.cam.ac.uldhandle/1810/270322 and in WO 2020/148251 A1.
At block 106, a scene energy is determined from the source data. A brightness and occupancy of the scene are extracted and used to determine the energy in the scene in the source data. For example, is the scene light or dark, and how much of the scene is occupied? In this block a number is determined for each hologram frame (or sub-frame in colour mode) that represents a desired illumination power to maintain a constant brightness for the image from the source data.
The scene energy may be used directly, however, in this example, the determined scene energy is quantized at block 108. Quantisation can provide several benefits. These include any or all of: reducing the resolution of the scene energy so it is easier to handle and process, ensuring that the scene energy corresponds to a value which can actually be output by a laser, and ensuring that that adjacent quantised scene energies result in a perceptible change in brightness (So that as the source scene occupancy and brightness changes, there is a corresponding visible change in illumination, and holographic ‘objects’ in the replay field seem to have consistent brightness).
The resulting quantised scene energy is then associated with the hologram data by adding it as metadata at block 110. The hologram data and laser metadata are then used to control the LCoS to display the holographic data while simultaneously controlling the power of the laser source using the scene energy metadata at block 112.
Control at block 112 can include hardware encoding, where the quantized scene energy values are encoded to into suitable values to drive the illumination sources to give the desired output power. This encoding may be performed by a dedicated hardware driver board that takes the input power values and converts to signals that can be interpreted by the illumination sources. There is no guarantee that the perceived image brightness is linear with the input power, so the hardware encoding may use a calibration to map quantised scene energies to specific input power values to control the illumination source.
In some examples, the control at block 112 includes receipt of data indicative of the temperature of the laser, for example from a temperature sensor integrated into the laser package or in contact with the package. Any suitable temperature sensor can be used, such as thermocouple. In this case, a plurality of calibration curves or look-up tables may be stored, each corresponding to a different laser temperature. A particular calibration curve or look-up table is selected based on the data indicative of temperature. The calibration curves or look-up tables may be determined from prior experimental testing of lasers of the same type, from data supplied by the manufacturer of the laser, and/or from a calibration of each laser as part of the manufacturing process.
In other examples, the control at block 112 may use predetermined data of laser output. As the laser is used its temperature will change, and so the output power will change over time for the same input. This temperature vs time relationship may be determined experimentally for a known input by measuring what happens to the temperature in the laser enclosure over time, such as by using a separate temperature probe. Display systems using the same model of laser will behave similarly, so the likely laser temperature of such can be determined by knowledge of the input and time of operation with that input. This can allow compensation for laser temperature without requiring a temperature sensor in the hardware, because the temperature is determined based on the input drive parameters and operation over time.
While the method of
At block 106, the scene energy data may be determined by considering pixel occupancy and pixel intensity. Pixel occupancy is determined by how many pixels are occupied (not black) in the source scene. Pixel intensity is determined from the colour values of each pixel to give a measure of the colour intensity of each occupied pixel. When the source data is expressed as RGB data, black pixels have red, green and blue encoded as zero. The sum of energy across all of the RGB values can therefore be used to determine the scene energy according to equation (1) below:
where i indexes the pixels in the source data and j={C1, C2, . . . , CN} are the allowed colours. The {cij} are the j-th colour component of the i-th pixel. The coefficients, {αj} are scaling constants that are dictated by the illumination source that will be generating each colour component in the holographic image. These coefficients are hardware specific and are determined based on the characteristics of the illumination sources used in a particular implementation. For example, the coefficients may be determined as a mean value for a particular model, part no. or SKU of illumination source from test data.
In one example, the coefficients {αj} are determined experimentally. A reference current was chosen for each of the Red, Green and Blue lasers. A hologram was calculated of a reference scene that was a white square, and the coefficients were adjusted until the perceived square in the replay field appeared to be white. The adjustment could be based on observation by a human or by measurement using a colorimeter, such as a passive display colorimeter or similar. It is possible that the colour will vary over the replay field. Some examples may repeat this measurement for a white square in different positions on the replay field and take an average, such as a mean value, of the results at the different positions.
The result of equation (1) may then be normalised to a range between E=0 (all black, nothing displayed) and E=1 (all pixels are white).
To give a more specific example, the source data received from Unity may comprise rendered scenes stored in memory as RGBZ data or point-cloud data. Each element, or pixel, is encoded as 32-bit RGBZ data. 32-bit RGBZ data comprises an 8-bit Red value, an 8-bit Green value, an 8-bit Blue value and an 8-bit depth value. For the purposes of the brightness control, the 8-bits of depth data are discarded to leave a 24-bit colour map that will be referred to as the source frame. Other examples can use different bit depths for the RGBZ data.
To apply equation (1), it is assumed that Unity renders the source frames at a resolution of M×N. This gives the total number of pixels over which the index i runs. For this implementation, there are red, green and blue laser sources and j is fixed to be in {0=Red, 1=Green, 2=Blue}. The calculated energy for each source frame is stored as metadata alongside the hologram computed from that source frame.
In another example, rather than summing the energy across all RGB values, the energy can be calculated for each component using equation 1a below:
This may be preferable when colour is reproduced by time-multiplexing individual colour components. The terms of the equation are the same as discussed. The energy can therefore vary by colour component rather than being set at the same value for all components.
In further examples, rather than using a square term as in equations 1 and 1a above, energy can be calculated in any suitable way, depending on the hardware and the source data used for display. For example, the calculation of the energy may incorporate a target gamma value of the display hardware, or otherwise account for gamma, such as by using predetermined data of the gamma.
More examples of the quantisation at block 108 will now be described. The calculated energy for a source frame, rendered to a typical displayed resolution can take many possible values. For a display of resolution M×N and colour bit depth k, the highest possible energy value is given by:
Highest possible energy value=M×N×22k Equation (2)
In fact, the true value used is M×N×(2k−1)2, because zero (black) is included in the 2k levels, but an approximation is made in (2) to make the equations in this discussion simpler. Thus, because the energy values are integers, there are also at most this many distinct energy values. For a high-resolution display with a large colour range, the number of possible energy values can become very large and can exceed the number of possible illumination levels that are available from the light source. The energy value is therefore quantised by defining a map, from the energy values to the allowed illumination source power values, [E1, E2, . . . , EM×N×2
The values a and b depend on the wavelengths of the light from the illumination sources, the physical properties of the illumination device, and the response characteristics of the display (LCoS in this case). Physical properties of the illumination device include a temperature response and inefficiencies from prolonged use. Due to this a and b values may be different for each illumination source in the display system.
A worked example of the quantisation will now be given. As discussed above, a Unity scene at a resolution of 1920×1080 with an RGB colour depth of 8-bits per pixel is supplied as the source frame. From equation (2), this yields approximately 229 possible energy values.
In this example, a phase modulating LCoS with resolution 1920×1080 is illuminated by RGB lasers. The laser control system accepts illumination information consisting of 29=512 distinct current modulation values and an additional 29=512 pulse width modulation values.
In an augmented reality (AR) display, the number of target levels to address in the quantisation can be further reduced by considering an average pixel occupancy of an AR scene. AR scenes are typically sparse, for example it may be assumed that only around 10% of the source frame pixels are occupied. The quantisation can therefore consider expected occupancy of the source data. In this AR example, the number of required energy levels is then at most (1920×1080×216)/10≈13.6 Billion.
In this example, a linear mapping is used as follows.
1. The maximum energy that can be achieved is calculated
Max=1920×1080×28×28/10 (as described above), this is set as b with a as zero.
2. The total energy for each sub-frame is found as, E=Σi ci2, for pixel intensities ci.
3. The scaled energy is then Escaled=E/Max.
Note that if the occupancy assumption is broken (the occupancy is greater than the maximum assumed, 10% in this case) then Escaled>1. Should this occur, the value of E is clipped to 1. To accurately display content with higher occupancy than 10%, the factor of 10 in the definition of Max should be reduced. This ensures the values are within the allowed range of laser powers available.
The illumination source may be driven by a driver circuit that takes as input the quantized illumination power levels and outputs a signal that sets the illumination source to emit light at the desired brightness. In this example the driver circuit outputs a signal which can be current modulated and/or pulse width modulated (PWM). With current modulation, the driver circuit sets the current to the illumination source, the power into the illumination source is proportional to the current and hence the output illumination power is also proportional to the current (the relationship may not be linear). With PWM, a voltage level input to the illumination source is predefined but the time the input is turned on is varied to vary the average input to the illumination source, again influencing the power in and hence the power output from the illumination source.
In this example, both PWM and current modulation are used, which may achieve consistent results across the range of input energy values. Note that the illumination source output power does not necessarily scale linearly with the energy value and so, for example, doubling the source energy value may not be equivalent to a doubling of the optical power. In that case, the non-linearity can be compensated by mapping the quantized scene energy values to the current and pulse length sent to the illumination sources.
The driver operates under one or more of the following constraints:
1. An input level of zero returns an output light power of zero.
2. A non-zero input level results in sufficient input power for a visible image in the hologram replay field.
3. For any input level, the output light power should not exceed the eye-safe limit.
4. In full colour operation, the driver balances the output power of the component illumination sources (red, green and blue) such that the output of all sources at full power is perceived to be pure white.
5. Each quantised level of the scene energy corresponds to visually distinguishable states.
An illumination source has a range of possible brightness it can produce in combination with a control circuit, based on input drive parameters such as a range of pulse lengths and current values. The pulse lengths are limited by the stabilisation time of the control circuit at the low end, and the framerate and stabilisation time of the Spatial Light Modulator (SLM) pixels at the high end.
In some examples, the scene energy is encoded as a combination of a current and PWM values to drive the illumination source. This expresses the scene energy in terms of driving parameters for the illumination source. No further conversion is necessary for display. When scene energy is encoded in this way and associated with hologram data, it can be used to set the power of an illumination source with good synchronisation with the display of the holographic data because no additional processing is required. Put another way, the scene energy may be encoded as values specific to a particular illumination source rather than in a more general format such as a scene energy which requires processing to derive the parameters to drive the illumination source.
When the hologram data is produced, this metadata is attached and then passed to an illumination source driver at the point the hologram is displayed. This driver ensures that the metadata will be used substantially immediately to drive the sources and avoid frame lags and maintain synchronicity between the target illumination and the hologram frame (or sub-frame in full colour operation).
While it is preferable to send the metadata to the illumination driver directly from the display driver, it is also possible to extract the metadata and send separately to the illumination driver. In that case a separate system to maintain synchronization between the displayed frame and the illumination can be provided.
For every frame (sub-frame in colour mode) the illumination driver performs a Cyclic Redundancy check (CRC) on the encoded scene energy data to ensure that valid data has been passed to the driver. If a CRC is failed or if data processing takes longer than a predefined threshold time, the illumination source is switched to the off state for that frame. This ensures that only valid signals trigger the illumination sources and so the output power cannot exceed the eye-safe limit.
In this example, current modulation and PWM are used for different ranges of control values, with PWM being used for values in a range 22 which is less than a predetermined threshold, τ, and current modulation being used for values in a range 24, greater than the predetermined threshold up to the predetermined maximum, μ.
For each source frame, the scene energy, E, is calculated as discussed above and a ratio, Λ, of source frame energy E to max energy μdetermined. If the ratio, A, is above the threshold value, τ, then the pulse width for PWM modulation is set to the maximum value and the current modulation is used to give the input power to the illumination source. If the ratio, Λ, is less than or equal to τ, then the current is set to a predetermined value and PWM is used. The predetermined value for current in PWM is chosen such that a minimum pulse width gives the required minimum visible output, and the maximum pulse width gives the threshold energy τ.
In one example, the value oft and the current for the PWM phase of control are determined empirically by first setting the PWM to a minimum value and then adjusting the current until a minimum brightness is achieved. This sets the value of the current for the PWM phase. Next, i is determined by using the determined current value and setting PWM to the maximum. In one implementation, the minimum value for PWM is zero, and the minimum current is set to be a value just above the lasing limit for the laser diode used. The minimum current can be experimentally determined or derived from the laser diode manufacturer datasheets. Typically, laser behaviour is non-linear and sensitive to temperature fluctuations and other factors close to the lasing limit so, in another implementation, rather than choosing the lowest observed current at which the laser diode beings to lase, a greater value of current is selected, for example a value at the bottom of a substantially linear area of laser response.
In some examples, the current modulation and PWM phases are inverted, so that the current modulation is used below the predetermined threshold and above the predetermined threshold PWM is used. Further examples may vary both the current modulation and PWM simultaneously over the whole or part of the control range.
It is possible that the illumination source may not have a linear response to the input power so that a scene energy is mapped to output powers to achieve the desired brightness. The mapping can compensate for or calibrate any non-linearity of the input source. One way to derive the mapping is as follows:
In some examples the value of ti can be chosen so that current modulation is used for a portion where the relationship between input and output power is linear and PWM is used for a portion where the relationship between input power and output power is non-linear. (Non-linear operation is typically the case in the lower ranges of output power). In that case the parameters for the control are found as follows:
Once the current modulation and PWM values have been determined, they are encoded in a data structure consisting of 3 bytes (24 bit) as follows
These are all packed into the first 24-bits of a 32-bit int which is then packed into the header of the hologram data object.
Holographic Display System with Illumination Source Power Control
An example system in which the methods and data structures described above can be implemented is depicted as a block diagram in
The configuration for the laser driver 34 is determined from the physical characteristics of the lasers and the LCoS display. At initialization, these parameters may be sent to the laser driver board from the computing system 30, for example over a USB link 40. Alternatively, the parameters may be predetermined and stored in firmware because they may remain the same for the lifetime of the display.
In operation, the computing system 30 calculates the holograms and associated laser driver or source energy metadata from received source data. The calculated hologram data and scene energy data is passed to the LCoS driver 38, for example using an HDMI or DisplayPort link 42. Next, the hologram is pushed to the LCoS display 36 while the laser metadata is sent to the laser driver 34, for example over an I2C bus, or serial link 44. The laser metadata determines the current and the pulse length used to drive the laser 32 and hence the output power. Synchronisation data is also exchanged between the LCoS driver and Laser Driver over link 46. For example, the synchronisation data can be a SYNC signal sent by the LCoS driver as a master device to the laser driver 34 as a slave device. The laser driver ensures that the lasers are activated in sync with the corresponding hologram frames on the LCoS display using the synchronisation data.
The data flow for the operation of the system may be as follows:
By determining and applying the source energy metadata the system of FIG. 6 can display holographic images with more consistent apparent brightness.
In an alternative example system, the system of
In other examples, display technologies other than LCoS are used, for example a spatial light modulator (SLM). Likewise, other connection protocols can be used instead of I2C, such as Serial Peripheral Interface (SPI). The constructions and principles of
While
Holographic display driver 54 receives Computer Generated Hologram (CGH) data from the program 52 along with scene energy metadata. For example, the CGH data may be sent over an image bus, such as one according to the DisplayPort, High-Definition Multimedia Interface (HDMI), Thunderbolt or USB standards. The scene energy metadata may be encoded as separate data associated with the CGH data and included in the data stream sent over the image bus along with the CGH data. In other examples, the scene energy metadata may be encoded in the image data itself and extracted by the holographic display driver prior to display. The scene energy metadata may be encoded into particular pixels of the CGH data in a way which produces little visible change to the reproduced CGH, such as by overwriting the least significant bit of particular pixels. The particular pixels are at predetermined positions and may be distributed over the frame area and/or located at a periphery of the frame to further reduce any visible change on the image when displayed.
Referring now to
A controller, or processor, 80 receives an input 82 of the CGH data for display, along with scene energy metadata. The controller 80 is configured to separate the scene energy metadata from the CGH data for display and provide the CGH data image to an SLM driver 84 for display on an SLM 86. The SLM driver 84 notifies the controller 80 via a signal 88 once the image is formed on the SLM. Next, the Controller 80 activates a laser 95, or other at least partially coherent light source, via a laser driver 94, to illuminate the SLM. In this way the controller synchronises the displayed CGH image with the correct illumination as defined by the scene energy metadata.
The controller of
The diffusion data may be included as metadata associated with each image. In some examples, the diffusion data is a value corresponding to a relative or absolute level of diffusion to apply. For example, a value of 0 might indicate no diffusion and a value of 1 might indicate that diffusion is active. Other examples of diffusion data are possible.
The controller 80 is configured to separate the diffusion data from the input images and provide the input image to an SLM driver 84 for display on an SLM 86. The SLM driver 84 notifies the controller 80 via a signal 88 once the image is formed on the SLM. Next, the Controller 80 provides a signal 90 to a diffusion control element 92 at the same time as activating a laser 94, or other at least partially coherent light source, to illuminate the SLM.
The diffusion control element 92 is configured to activate and deactivate a diffuser 96 depending on a required speckle reduction in the output image according to the diffusion data. In some embodiments, the diffusion control element 92 is also configured to control a level of diffusion provided by the diffuser. For example, the diffusion control element may control a level of diffusion by selectively activating an ultrasonic mirror and/or adjusting a focus point. Furthermore, if the diffuser is binary, or otherwise has discrete states, the diffusion control element may selectively activate and deactivate the diffuser or switch between states during display of one image according to the desired diffusion amount, such as by using Pulse Width Modulation.
This construction of
Controller 80 is implemented by a Field Programmable Gate Array (FPGA) in
Some examples may combine the functional blocks depicted in
High-Definition Multimedia interface (HDMI) is used to supply image data to the controller in
In some examples, the scene energy metadata and/or the diffusion data is independent of the display hardware and the controller 80 translates the scene energy metadata and the diffusion data to appropriate hardware operation. For example, the diffusion data may be set as predetermined values according to a required level of diffusion and this can be realised using the hardware via the controller 80 and/or the diffusion control 92. Similarly, the scene energy metadata may be independent of an illumination source and determined by the controller 80 based on predetermined characteristics of the laser 94.
Although
In some examples, the system of
Some examples include an intensity monitoring system, comprising a light intensity sensor mounted within the optical path of the holographic display. The illumination control system can provide feedback control of the output illumination power and/or ensure that the output brightness is within the expected tolerances for the input source frames. An advantage of such feedback control is reduced temperature sensitivity. A block diagram of an example illumination monitoring system is depicted in
In one example, the display 64 is a phase modulating LCoS display, the illumination source 68 is an RGB laser source and the photosensor 62 is a photo-transistor mounted inside an optical engine.
The above embodiments are to be understood as illustrative examples of the invention. Further embodiments of the invention are envisaged. For example, although the discussion has focused on source RGB data, the concepts can be applied to any colour space used to represent an image. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015789.7 | Oct 2020 | GB | national |
This application is a continuation under 35 U.S.C. § 120 of International Application No. PCT/GB2021/052566, filed Oct. 5, 2021, which claims priority to GB Application No. GB 2015789.7, filed Oct. 6, 2020, under 35 U.S.C. § 119(a). Each of the above-referenced patent applications is incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/GB2021/052566 | Oct 2021 | US |
| Child | 18192118 | US |
