Patent Application
20220394221
The present invention generally relates to a structured-light scanning system, and more particularly to a structured-light scanning system with thermal compensation.
Structured-light scanning is the process of projecting a pattern of light onto a scene. The reflected pattern is captured by a camera, and then processed, for example, by triangulation, to reconstruct a three-dimensional or depth map of the objects in the scene. The structured-light scanning may, for example, be adapted to object detection for detecting objects of a certain class in digital images and videos. Specifically, the structured-light scanning may be adapted to face detection, which is a specific case of object detection, in mobile devices such as cellphones for detecting frontal human faces for security reasons.
However, the components of a structured-light scanning system are liable to variations, drifts or deformations due to temperature change. Such variations may distort a projected pattern of light, and thus reduce detection accuracy. For example, wavelength of a light source may vary, effective focal length of a lens may drift, and a printed circuit board may be deformed due to temperature change.
A need has thus arisen to propose a novel scheme to compensate thermal effects of a structured-light scanning system.
In view of the foregoing, it is an object of the embodiment of the present invention to provide a structured-light scanning system capable of compensating thermal effects.
According to one embodiment, a structured-light scanning system with thermal compensation includes a structured-light projector, an image sensor and a digital processing unit. The structured-light projector generates a predetermined projected pattern of light, which is then projected onto and reflected from an object, thereby resulting in a reflected pattern of light. The image sensor captures the reflected pattern of light. The digital processing unit generates a depth map according to the reflected pattern of light and a compensated projected pattern associated with a current temperature. In another embodiment, the digital processing unit generates a depth map according to the reflected pattern of light, a reference projected pattern and a displacement map associated with a current temperature.
In the embodiment, the structured-light scanning system with thermal compensation (system hereinafter) 100 may include a structured-light projector 11 configured to generate a predetermined projected (or ground truth) pattern of light. The projected pattern of light is then projected onto and reflected from an object (not shown), resulting in a reflected pattern of light.
The system 100 of the embodiment may include an image sensor 12, such as a camera, configured to capture the reflected pattern of light.
In one embodiment, the system 100 may include a holder 13 (also called immobilizer), such as a printed circuit board, configured to hold the structured-light projector 11 and the image sensor 12 such that relative positions of the structured-light projector 11 and the image sensor 12 may be maintained.
According to one aspect of the embodiment, the system 100 may include a thermal sensor 14 (also called temperature sensor) configured to measure current (ambient) temperature, according to which thermal compensation may be performed. In one embodiment, the thermal sensor 14 may be disposed in or adjacent to the structured-light projector 11 for the reasons that the structured-light projector 11 may ordinarily suffer temperature rise and fall. Generally speaking, however, the thermal sensor 14 may be disposed everywhere in the system 100 for measuring the ambient temperature thereof.
According to another aspect of the embodiment, the system 100 may include a memory device 15, such as a flash memory or read-only memory, configured to provide a (predetermined) compensated projected pattern (or ground truth map) associated with the current temperature. In one embodiment, a plurality of compensated projected patterns associated with different temperatures are stored in the memory device 15. In another embodiment, a reference projected pattern associated with a reference temperature (e.g., room temperature) and a plurality of displacement maps (DMs) (representing dot shift between corresponding compensated projected patterns and the reference projected pattern) associated with different temperatures are stored in the memory device 15.
The system 100 of the embodiment may include a digital processing unit 16 configured to generate a (compensated) depth map according to the reflected pattern of light (from the image sensor 12) and a compensated projected pattern associated with the current temperature. Specifically, the digital processing unit 16 may include a buffer 161, such as a random-access memory, configured to temporarily store the compensated projected pattern associated with the current temperature.
The digital processing unit 16 may include a depth decoder 162 configured to generate the (compensated) depth map according to the compensated projected pattern (associated with the current temperature) and the reflected pattern (from the image sensor 12). Conventional depth decoding techniques may be adopted to obtain the depth map, details of which are thus omitted for brevity.
The digital processing unit 16 may include a controller 163 configured to control operations of the buffer 161 and the depth decoder 162 (as denoted by the dashed lines), configured to coordinate the structured-light projector 11 and the image sensor 12 (as denoted by the dashed lines), and particularly configured to controllably obtain the compensated projected pattern (associated with the current temperature) from the memory device 15 according to the current temperature received from the thermal sensor 14.
In step 21, the system 100 is currently subject to a controlled ambient temperature, and, in step 22, a corresponding compensated projected pattern is obtained such that a (predetermined) objective depth map may be generated by the depth decoder 162. In step 23, the compensated projected pattern is stored in the memory device 15. In step 24, the ambient temperature is controllably changed (for example, increased), and the flow goes back to step 21 if the changed temperature is still within a predetermined range (step 25). Accordingly, a plurality of compensated projected patterns associated with different temperatures are stored in the memory device 15. It is noted that, if the variation of the compensated projected patterns is device-independent, the calibration stage may be performed only once, otherwise the calibration stages should be respectively performed with respect to individual devices.
In step 26, the current temperature (provided by the thermal sensor 14) is read by the controller 163. Next, in step 27, a corresponding compensated projected pattern associated with the current temperature is retrieved from the memory device 15 and is temporarily stored in the buffer 161. As exemplified in
In step 31, the system 100 is subject to a reference temperature, and, in step 32, a corresponding reference projected pattern is obtained such that a (predetermined) objective depth map may be generated by the depth decoder 162. The reference projected pattern is then stored in the memory device 15. In step 33, the system 100 is currently subject to a controlled ambient temperature, and, in step 34, a corresponding displacement map (DM) between a corresponding compensated projected pattern and the reference projected pattern is obtained. In step 35, the displacement map (DM) is stored in the memory device 15. In step 36, the ambient temperature is controllably changed (for example, increased), and the flow goes back to step 33 if the changed temperature is still within a predetermined range (step 37). Accordingly, the reference projected pattern and a plurality of displacement maps (DMs) associated with different temperatures are stored in the memory device 15.
In step 38, the current temperature (provided by the thermal sensor 14) is read by the controller 163. Next, in step 39, a reference projected pattern associated with a reference temperature and a displacement map associated with the current temperature are obtained from the memory device 15.
Subsequently, in step 40, the depth decoder 162 generates the (compensated) depth map according to the reference projected pattern, the displacement map (associated with the current temperature) and the reflected pattern (from the image sensor 12). Specifically, a compensated projected pattern may be generated by warping the reference projected pattern with the displacement map, for example, by an interpolator 164 (for example, disposed in the digital processing unit 16) as shown in
Subsequently, the depth map may then be generated (by the depth decoder 162) according to the compensated projected pattern and the reflected pattern (from the image sensor 12). It is noted that, when there is no displacement map associated with a temperature matching with the current temperature, a displacement map may be generated (or estimated) by performing interpolation on the two displacement maps associated with temperatures (just) larger and (just) less than the current temperature, respectively. For example, in the example as shown in
Although specific embodiments have been illustrated and described, it will be appreciated by those skilled in the art that various modifications may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.
