Patent Grant
10469827
This application is a U.S. National Phase of International Patent Application No. PCT/JP2014/079137 filed on Oct. 31, 2014, which claims priority benefit of Japanese Patent Application No. JP 2013-273231 filed in the Japan Patent Office on Dec. 27, 2013. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.
The present disclosure relates to an image processing device and an image processing method.
In recent years, as a method of generating a three-dimensional image of an object, a system like the one disclosed in Patent Literature 1 below which irradiates infrared (IR) rays and obtains depth information of an object to thereby generate a three-dimensional image, has been widely recognized.
Specifically, utilizing the characteristics of infrared rays that they are invisible to the human eye, the system irradiates a pattern onto the surface of the object using infrared rays and captures an image with an infrared camera, thereby obtaining the surface shape of the object in more detail and more accurately based on the captured image.
Patent Literature 1: JP 2008-537190 A
However, there is an issue that irradiating the infrared pattern onto the object may cause loss of information, such as a contour of the object and a pattern on the surface of the object which are features the object originally has, and an accurate object shape may not be obtained depending on conditions.
Although there is also a system that additionally uses a visible light camera for capturing texture, in that case, in addition to the infrared camera for detecting depth, other types of cameras are required, and an issue of increase in system cost arises. Moreover, there arises an issue that the system becomes extremely large due to increase of signal processing for accurately matching positions of captured images captured by a plurality of types of cameras.
Furthermore, there is an issue that, if a visible light image projected by a projector overlaps the object, an accurate object shape cannot be obtained even with the visible light camera.
Accordingly, the present disclosure proposes an image processing device and an image processing method capable of obtaining a pattern-irradiated infrared image and a pattern-irradiation-free infrared image to obtain more accurate depth information by controlling an infrared pattern irradiation timing.
According to the present disclosure, there is provided an image processing device including: a pattern irradiation unit that irradiates an infrared pattern onto a surface of a target object; and an infrared image capturing unit that captures an infrared image. The pattern irradiation unit performs irradiation at a predetermined timing corresponding to an infrared image capturing unit's image capturing timing. The infrared image capturing unit obtains a pattern-projected infrared image in which the pattern irradiated by the pattern irradiation unit is projected on the target object, and a pattern-free infrared image in which the pattern is not projected on the target object.
According to the present disclosure, there is provided an image processing method including: irradiating an infrared pattern onto a surface of a target object by a pattern irradiation unit; and capturing an infrared image by an infrared image capturing unit. The pattern irradiation unit performs irradiation at a predetermined timing corresponding to an infrared image capturing unit's image capturing timing. The infrared image capturing unit obtains a pattern-projected infrared image in which the pattern irradiated by the pattern irradiation unit is projected on the target object, and a pattern-free infrared image in which the pattern is not projected on the target object.
As described above, according to the present disclosure, by controlling the timing of irradiating an infrared pattern, it becomes possible to obtain a pattern-irradiated infrared image and a pattern-irradiation-free infrared image for obtaining more accurate depth information.
Moreover, the effects described above are not necessarily limited, and along with or instead of the effects, any effect that is desired to be introduced in the present specification or other effects that can be expected from the present specification may be exhibited.
Hereinafter, embodiments of the present disclosure will be described in detail with reference to the appended drawings. In this specification and the drawings, elements that have substantially the same function and structure are denoted with the same reference symbols, and redundant description is omitted.
In addition, the description will be made in the following order.
1. Image Processing Device According to First Embodiment
2. Image Processing Device According to Second Embodiment
3. Image Processing According to Third Embodiment
4. Summary
<1-1. Configuration>
First, a configuration of an image processing device 10-1 according to the first embodiment of the present disclosure will be described with reference to
As shown in
(Timing Control Unit)
The timing control unit 101 is composed of a control circuit that outputs pulses to the IR stereo camera 102, the IR random pattern irradiation unit 103, and the IR irradiation unit 104. The timing control unit 101 controls the IR random pattern irradiation unit 103 and the IR irradiation unit 104 to perform irradiation, for example, alternately at the timings of image capturing by the IR stereo camera 102, thereby enabling random pattern-irradiated images and pattern-free images to be captured alternately. Specific operation control of the image capturing and irradiation timings by the timing control unit 101 will be described later with reference to
(IR Stereo Camera)
The IR stereo camera 102 is an example of an infrared image capturing unit composed of two cameras capable of capturing only an IR (infrared) region. Specifically, the IR stereo camera 102 can capture an image of only the IR region, for example, by mounting a visible light cut filter. In addition, the two cameras constituting the IR stereo camera 102 are positioned with an interval for generating a sufficient parallax. In these two cameras, because the shutters are released (images are captured) at the rising edge of an IR stereo camera control pulse 201 (see
(IR Random Pattern Irradiation Unit)
The IR random pattern irradiation unit 103 is an example of a pattern irradiation unit that irradiates an infrared (IR) pattern onto the surface of a target object, and it irradiates an IR random pattern. Here,
As shown in
The light emission control unit 301 controls light emission by causing electric current to flow to the IR light source 302 for a certain period in accordance with the rising of an IR random pattern irradiation unit control pulse (see
The IR light source 302 is an infrared ray emission unit and is implemented with an infrared LED (Light Emitting Diode), for example. While the light emission control unit 301 causes electric current to flow, the IR light source 302 irradiates infrared wavelength (e.g. wavelength around 850 nm). The infrared rays irradiated from the IR light source 302 pass through the random pattern mask 303.
The random pattern mask 303 is implemented with a metal mesh plate which has been subjected to minute and random drilling processing by electroforming, for example. The random mesh can be implemented, for example, by positioning points randomly and then generating a Voronoi diagram using the points as generators. Although, in addition, a minute random pattern can also be projected, for example, by printing a random pattern or dots on a film or using etching on glass, infrared rays are enabled to pass through the random pattern without being attenuated using the minute drilling processing by electroforming.
The projection lens 304 causes an IR random pattern 310 formed by infrared rays that passed through the random pattern mask 303 to be projected onto the target object. Although, in the example shown in
(IR Irradiation Unit)
The IR irradiation unit 104 has the function of irradiating pattern-free infrared rays. Because a specific structure of the IR irradiation unit 104 is obtained by removing the random pattern mask 303 from the IR random pattern irradiation unit 103 shown in
The IR irradiation unit 104 is not an essential component in the image processing device 10-1 according to the present embodiment, and it can be omitted if there is ambient light that includes infrared rays moderately, such as a light bulb. In addition, because the IR irradiation unit 104 do not need to irradiate the IR random pattern 310, the optical lens, such as the projection lens 304, can also be omitted if there is an environment in which infrared rays can be irradiated uniformly, such as when the distance between the target object (subject) and the IR irradiation unit 104 is sufficiently close.
(Random Pattern LR Image Storage Unit)
The random pattern LR image storage unit 105 stores two IR random pattern-irradiated images (L image and R image) captured by the IR stereo camera 102 composed of two infrared cameras (left camera, right camera) while the IR random pattern irradiation unit 103 projects the IR random pattern 310.
(Stereo Image Matching Unit)
The stereo image matching unit 107 is a depth calculation unit that generates a depth image having depth information using the LR images stored in the random pattern LR image storage unit 105. Although various conventionally known techniques can be used as a method of generating a depth image, a block matching method is used here as an example. Hereinafter, the method will be described in detail with reference to
Specifically, when, for example, image matching is performed per block of 8×8, a block that is close to a block 705 of 8×8 pixels having a target pixel 704 as a center in the R image 701 captured by the right camera is searched from the L image 702, to thereby obtain a parallax. More specifically, the stereo image matching unit 107 performs image comparison between the block 705 of 8×8 in the random pattern R image and the block 707 of 8×8 in the random pattern L image, for each of all horizontal pixels in a search range 706 in the L image 702. For example, when the search range 706 has 64 pixels, block image matching is performed 64 times for each pixel of the R image 701 which is an input image, and this is performed for all the pixels of the R image 701, which is the input image.
Thus, as the stereo image matching unit 107 performs matching of the right and left images on a per block basis, even a target object having no features on the surface (flat with no pattern) such as, for example, a white box which has not been able to be subjected to matching so far, can be subjected to matching accurately by using the IR random pattern-irradiated images.
Further, the right camera and the left camera constituting the IR stereo camera 102 are fixed, and the distance (baseline length) between the cameras is fixed. Therefore, the depth can be obtained based on the parallax obtained by block matching and the baseline length.
By performing the above-described matching processing on all the pixels, a depth image of the entire screen can be obtained.
(Depth Image Storage Unit)
The depth image storage unit 108 stores the depth image obtained by the stereo image matching unit 107.
(IR Image Storage Unit)
The IR image storage unit 106 stores an IR image (pattern-free infrared image) captured by the IR stereo camera 102 while the IR irradiation unit 104 performs irradiation. Here, the IR image stored in the IR image storage unit 106 may be an image captured by either the left camera or the right camera constituting the IR stereo camera 102. Specifically, only the L image captured by the left camera or only the R image captured by the right camera may be used, or, alternatively, the LR images captured by the both cameras may be stored and used, according to intended use.
(Image Recognition Unit)
The image recognition unit 109 performs conventionally known image recognition processing (e.g. edge detection processing with a differential filter, etc.) including recognition of the contour of the target object and performing character recognition, based on the IR image (pattern-free infrared image) stored in the IR image storage unit 106.
In particular, in the case of image processing like character recognition in which lines constituting a character are important information, if the image stored in the random pattern LR image storage unit 105 on which the random pattern is projected is used, the random pattern causes generation of an unintended shadow on the character, and resulting in decline of the recognition rate. In addition, if the random pattern-irradiated image is used, not only the contour of the subject but also the contour of the random pattern are detected in the edge detection processing such as differential filtering, and the contour of the subject cannot be obtained accurately. Therefore, the image recognition unit 109 uses the IR image stored in the IR image storage unit 106 on which no random pattern is irradiated, thereby recognizing characters written on the surface of the target object, such as a title of a book, and the contour of the target object accurately.
Further, when the visible light camera is used in image processing in which the contour of the object is important information to recognize the shape of the target object, it may cause decline of the recognition rate of the contour. For example, if a visible light image is projected from the projector onto a table, etc., and the user performs operation input manually for the image projected on the table, a position of the hand, the shape of the hand, and change of the hand shape, etc. are used as operation input information. In this case, if the visible light camera is used, the projected image is reflected on the surface of the hand which is the target object, and therefore, the recognition rate of the shape and contour of the hand declines. Accordingly, the image recognition unit 109 according to the present embodiment uses the IR image captured by the IR stereo camera 102 implemented with the infrared cameras, thereby recognizing the shape and contour of the hand accurately without being affected by the projected visible light image reflected on the hand surface.
(Recognition Result Storage Unit)
The recognition result storage unit 110 stores the result recognized in the image recognition unit 109.
The image processing device 10-1 having the above-described configuration can recognize and output a three-dimensional shape (depth image) of the target object (subject), and characters written on the surface of the target object and the contour of the target object. Next, the operation control of the image capturing and irradiation timings by the timing control unit 101 will be described specifically with reference to
<1-2. Operation>
In addition, the control pulses 202 and 203 are control pulses transmitted to the IR random pattern irradiation unit 103 and the IR irradiation unit 104, respectively, and the irradiation units are designed to turn on the IR light source for a certain period at the rising edges of these pulses. The control timings of the random pattern irradiation unit 103 and IR irradiation unit 104 are set to cause light emission alternately in the half cycle of the IR stereo camera control pulse 201, as shown in
In doing so, the IR stereo camera 102 can obtain an image of the target object on which the random pattern is irradiated (pattern-projected infrared image) and an image of the target object on which IR without any patterns is irradiated (pattern-free infrared image) alternately in time.
The irradiation control timings of the random pattern irradiation unit 103 and the IR irradiation unit 104 are not limited to the example shown in
In any case, the timing control unit 101 can enable the IR stereo camera 102 to obtain an image of the target object on which the random pattern is irradiated (pattern-projected infrared image) and an image of the target object on which pattern-free IR irradiation is performed (pattern-free infrared image) at different timings in time.
After the images are obtained, as described above, the stereo image matching unit 107 generates the depth image while the image recognition unit 109 performs image processing, such as character recognition.
<1-3. Supplement>
Next, supplemental explanation of the present embodiment will be made.
Although, in the above-described embodiment, the depth information is detected using the IR stereo camera 102, the present embodiment is not limited thereto, and the depth information can also be detected using a monocular IR camera.
In this case, the image processing device 10-1 stores the random pattern for IR irradiation in advance and performs block matching between the stored random pattern and an image captured by one IR camera, to thereby calculate the depth. Hereinafter, the description will be made with reference to
The image processing unit 10-1 calculates the depth using the known distance d based on coordinate values of the random pattern subjected to matching on the image captured by the IR camera 112, and in particular, based on a coordinate position of the random pattern reflected on the target object S. More specifically, the image processing device 10-1 estimates a coordinate position of the pattern which could be reflected on the wall surface if the target object S does not exist, and detects the depth information in accordance with a parallax between the estimated coordinate position and the coordinate position of the random pattern reflected on the target object S.
<1-4. Effect>
As described above, with the first embodiment, it is possible to obtain accurate depth information by irradiating the infrared random pattern onto the object, even if the object has no feature on its surface. Further, by also obtaining the infrared image on which the random pattern is not irradiated, it is also possible to precisely perform image processing which does not require the random pattern or which is affected by the random pattern in terms of accuracy, etc.
Further, because the camera is used in a time sharing manner for obtaining the depth and for processing the image, there is no need of adjusting the positions of two cameras, and the two resulting positions can be matched easily.
Further, by turning on the IR random pattern irradiation unit 103 and the IR irradiation unit 104 alternately so as to synchronize with the shutter timings (timings of image capturing), the light emission intervals of the irradiation units are extended, and the duty ratio of light emission can be reduced. Then, as the duty ratio of light emission can be reduced, it becomes possible to increase the amount of electric current flowed at once and provide brighter light emission. In doing so, the effects that are effective in image processing after image capturing, such as enabling increase of the shutter speed or enabling reduction of gain to suppress noise, can be obtained.
Next, an image processing device 10-2 according to the second embodiment will be described with reference to
<2-1. Configuration>
Because the random pattern LR image storage unit 105, the stereo image matching unit 107, the depth image storage unit 108, the image recognition unit 109, and the recognition result storage unit 110 are similar to those in the first embodiment described with reference to
(Timing Control Unit)
The timing control unit 401 is composed of a control circuit that outputs pulses to the IR stereo camera 402, the IR camera 403, the IR random pattern irradiation unit 404, and the IR irradiation unit 405. The control of the image capturing and irradiation timings by the timing control unit 401 will be described later with reference to
(IR Random Pattern Irradiation Unit)
Although the structure of the IR random pattern irradiation unit 404 is similar to the IR random pattern irradiation unit 103 of the first embodiment described with reference to
(IR Irradiation Unit)
Although the structure of the IR irradiation unit 405 is also similar to the IR irradiation unit 104 according to the first embodiment, it differs from the IR irradiation unit 104 in that a light source is selected so that the wavelength of the IR light source becomes, for example, 750 nm.
(IR Stereo Camera)
The IR stereo camera 402 has a bandpass filter, a low-pass filter, or a high-pass filter that allows, for example, the wavelength around 850 nm in the IR (infrared) region to pass through, and the IR stereo camera 402 has a structure in which light having the wavelength other than 850 nm has the lowered sensitivity and is not easily exposed. In doing so, the IR stereo camera 402 can capture an image of the target object on which the random pattern irradiated from the IR random pattern irradiation unit 404 is superposed.
(IR Camera)
The IR camera 403 has a bandpass filter, a low-pass filter, or a high-pass filter that allows, for example, a wavelength around 750 nm in the IR (infrared) region to pass through, and the IR camera 403 has a structure in which light having the wavelength other than 750 nm has the lowered sensitivity and is not easily exposed. In doing so, because the IR camera 403 cuts off the wavelength irradiated from the random pattern irradiation unit 404, it is possible to capture a (pattern-free) image in which the random pattern is not superposed on the subject.
The IR camera 403 can be implemented with at least one infrared camera.
(IR Image Storage Unit)
The IR image storage unit 106a stores one image captured by the IR camera 403 implemented with at least one infrared camera.
The image processing device 10-2 having the above-described configuration can recognize and output the three-dimensional shape (depth image) of the target object (subject), and characters written on the surface of the target object and the contour of the target object, like the first embodiment.
<2-2. Operation>
Next, the operation control of the image capturing and irradiation timings by the timing control unit 401 will be described in detail with reference to
The IR stereo camera 402 and the IR camera 403 are designed to release their shutters at the rising edges of the sync pulses. In addition, the IR random pattern irradiation unit 404 and the IR irradiation unit 405 are designed to turn on the IR light source for a certain period at the rising edges of the sync pulses.
In other words, the timing control unit 401 according to the present embodiment controls the timings of image capturing by the IR stereo camera 402 and the IR camera 403 and the timings of irradiation by the IR random pattern irradiation unit 404 and the IR irradiation unit 405 to be the same timings.
Even if the IR stereo camera 402 captures the image at the timing of irradiating the random pattern from the IR random pattern irradiation unit 404, the IR camera 403 has the filter for cutting off the irradiation wavelength of the random pattern as described above, and therefore, it is possible to capture a (pattern-free) image in which the random pattern is not superposed on the subject. In doing so, it is possible to obtain an infrared image having a pattern and an infrared image having no pattern, without generating a time lag.
After obtaining the images, like in the first embodiment, the stereo image matching unit 107 generates a depth image, while the image recognition unit 109 performs image processing, such as character recognition.
<2-3. Effect>
With the second embodiment having the above-described configuration, like the first embodiment, it is possible to obtain the effects that accurate depth information can be obtained even if the object has no feature on its surface, and that image processing, which does not require the random pattern or which is affected by the random pattern in terms of accuracy, etc., can be performed precisely.
Further, in the second embodiment, unlike the first embodiment, the IR random pattern is irradiated in a predetermined wavelength, and images are captured such that the IR stereo camera 402 on which an optical filter through which the predetermined wavelength passes is installed is synchronized with the IR camera 403 on which an optical filter which cuts off the predetermined wavelength is installed. This causes the effect that, without producing a time lag between the image on which the random pattern is projected and the image on which no random pattern is projected, both image processing results are synthesized and associated more easily.
Next, an image processing device 10-3 according to the third embodiment will be described with reference to
<3-1. Configuration>
Because the timing control unit 101, the IR stereo camera 102, the IR random pattern irradiation unit 103, IR irradiation unit 104, the random pattern LR image storage unit 105, the IR image storage unit 106, the image recognition unit 109, and the recognition result storage unit 110 are similar to those in the first embodiment described with reference to
(Stereo Image Matching Unit)
The stereo image matching unit 107a generates a depth image based on the LR images of the target object stored in the random pattern LR image storage unit 105 on which the random pattern is projected. Here, although various methods of generating the depth image are possible, if the block matching method which has been described with reference to
In consideration of this, in order to reduce the amount of computation in block matching, it is possible to carry out a method of reducing the size of the R image 701 and L image 702, which are the input images, or a method of reducing the number of matching times such that the number of matching times are decimated by performing matching per two pixels or four pixels horizontally and per two pixels or four pixels vertically, but not on all the pixels in the R image 701. However, although these methods have great effect in reduction of the amount of computation, they have an issue that the resolution of obtained depth information is degraded. Hereinafter,
On the other hand, in the (low-resolution) depth image 802a obtained when an image of the hand, which is the same target object, is captured and the amount of computation is reduced, the shape of the fingers cannot be identified. The depth image 802a is a low-resolution depth image generated when the method of reducing the size of the R image 701 and the L image 702, which are the input images, is used as the method of reducing the amount of computation. Further, in the example shown in
Thus, when the amount of computation is reduced, the shape of the target object cannot be identified, and therefore, it becomes problematic in particular when the shape of the fingers is important information. Accordingly, in the present embodiment, the below-described high-resolution depth calculation unit 609 synthesizes the contour image of the IR image with the low-resolution depth image, to thereby generate a high-resolution depth image.
(Depth Image Storage Unit)
The depth image storage unit 108a stores the low-resolution depth image which is generated in the stereo image matching unit 107a and in which the amount of computation is reduced.
(High-Resolution Depth Calculation Unit)
The high-resolution depth calculation unit 609 obtains, from the recognition result storage unit 110, the contour information of the target object recognized based on the IR image. As described in the first embodiment, the recognition result storage unit 110 stores the contour information of the target object recognized by the image recognition unit 109 based on the IR image. In addition, the high-resolution depth calculation unit 609 obtains, from the depth image storage unit 108a, the low-resolution depth image generated based on the LR images, which have been reduced to reduce the amount of computation.
Then, the high-resolution depth calculation unit 609 generates a high-resolution depth image based on the contour information of the target object and the low-resolution depth image. Hereinafter, the specific description will be made with reference to
On the other hand, the low-resolution depth image 802a is a depth image generated by the stereo image matching unit 107a by reducing the L image and the R image, which are the input images, to reduce the amount of computation. The L image and the R image, which are the inputs, are the IR images with the random pattern captured by the IR stereo camera 102 at the time of irradiation of the random pattern by the IR random pattern irradiation unit 103. The depth image 802a generated by the stereo image matching unit 107a is stored in the depth image storage unit 108a.
Then, the high-resolution depth calculation unit 609 obtains and synthesizes the low-resolution depth image 802a stored in the depth image storage unit 108a and the contour image 902 of the IR image stored in the recognition result storage unit 110. Specifically, the high-resolution depth calculation unit 609 enlarges the low-resolution depth image 802a as shown in
In a synthesized image 903 shown in
The depth image 904 having high-definition contour information generated by the high-resolution depth calculation unit 609 is stored in the high-resolution depth image storage unit 610.
<3-2. Operation>
The configuration of the image processing device 10-3 according to the present embodiment and generation of the high-resolution depth image have been described specifically. The control of the image capturing and irradiation timings by the timing control unit 101 of the image processing device 10-3 is similar to the first embodiment described with reference to
<3-3. Effect>
As described above, with the third embodiment, it is possible to reduce the amount of computation in depth calculation, for example, by reducing the input image or performing decimation, and obtain a high-resolution depth image.
In addition, like the first embodiment, by obtaining the depth information based on the image on which the IR random pattern is projected, it is possible to obtain the depth information from even a flat subject having no pattern. Further, by using the image on which the IR random pattern is not projected, it is possible to perform edge processing of the target object accurately.
The configuration of the third image processing device 10-3 according to the third embodiment is not limited to the example shown in
As described above, the image processing devices 10-1 to 10-3 according to the embodiments of the present disclosure control the timing of infrared pattern irradiation thereby obtaining the infrared image on which the pattern is irradiated and the infrared image on which the pattern is not irradiated, to obtain more accurate depth information.
The preferred embodiment(s) of the present disclosure has/have been described above with reference to the accompanying drawings, whilst the present disclosure is not limited to the above examples. A person skilled in the art may find various alterations and modifications within the scope of the appended claims, and it should be understood that they will naturally come under the technical scope of the present disclosure.
For example, it is also possible to create a computer program for enabling hardware integrated in the above-described image processing devices 10-1 to 10-3, such as a CPU, ROM, and RAM, to perform the functions of the image processing devices 10-1 to 10-3. A computer readable storage medium in which the computer program is stored is also provided.
Further, generation of the depth information based on the IR image captured by the monocular IR camera 112 and the random pattern stored in advance can also be performed in the second and the third embodiments in a similar manner to that described in the supplement of the first embodiment.
In addition, the effects described in the present specification are merely illustrative and demonstrative, and not limitative. In other words, the technology according to the present disclosure can exhibit other effects that are evident to those skilled in the art along with or instead of the effects based on the present specification. Additionally, the present technology may also be configured as below.
(1) An image processing device including:
a pattern irradiation unit that irradiates an infrared pattern onto a surface of a target object; and
an infrared image capturing unit that captures an infrared image, wherein the pattern irradiation unit performs irradiation at a predetermined timing corresponding to an infrared image capturing unit's image capturing timing, and
the infrared image capturing unit obtains a pattern-projected infrared image in which the pattern irradiated by the pattern irradiation unit is projected on the target object, and a pattern-free infrared image in which the pattern is not projected on the target object.
(2) The image processing device according to (1), wherein the pattern irradiation unit irradiates simultaneously with predetermined interval of the infrared image capturing unit's image capturing timings.
(3) The image processing device according to (2), wherein the pattern irradiation unit irradiates at odd or even timings of the infrared image capturing unit's image capturing timings.
(4) The image processing device according to (2) or (3), further including
an infrared irradiation unit having no pattern, wherein,
among the infrared image capturing unit's image capturing timings, the infrared irradiation unit irradiates a pattern-free infrared ray at a timing different from the pattern irradiation unit's irradiation timing.
(5) The image processing device according to (4), wherein the infrared irradiation unit irradiates the pattern-free infrared ray alternately with the pattern irradiation unit.
(6) The image processing device according to any one of (1) to (5), further including
a first depth calculation unit that performs block matching on the basis of the pattern-projected infrared image captured by the infrared image capturing unit and a pattern preliminarily stored and irradiated by the pattern irradiation unit, to calculate depth information of the target object.
(7) The image processing device according to any one of (1) to (5), wherein the infrared image capturing unit includes two or more infrared image capturing units, and
the image processing device further comprises
(8) The image processing device according to (1), wherein
the pattern irradiation unit irradiates the pattern using an infrared ray of a predetermined wavelength,
the infrared image capturing unit includes a first infrared image capturing unit and a second infrared image capturing unit,
the first infrared image capturing unit includes at least one of a bandpass filter, a low-pass filter, and a high-pass filter that allows infrared rays around a wavelength irradiated by the pattern irradiation unit to pass through, and
the second infrared image capturing unit includes at least one of a bandpass filter, a low-pass filter, and a high-pass filter that cuts off an infrared ray having the wavelength irradiated by the pattern irradiation unit.
(9) The image processing device according to (8), further including
an infrared irradiation unit having no pattern, wherein
the second infrared image capturing unit includes a bandpass filter, a low-pass filter, or a high-pass filter that allows infrared rays around the wavelength irradiated by the infrared irradiation unit to pass through.
(10) The image processing device according to (9), wherein timings of image capturing by the first infrared image capturing unit and the second infrared image capturing unit and timings of irradiation by the infrared irradiation unit and the pattern irradiation unit are all the same timings.
(11) The image processing device according to any one of (1) to 5 and 8 to (10), further including
a third depth calculation unit that performs block matching requiring reduced amount of computation on the basis of the pattern-projected infrared image captured by the infrared image capturing unit and a pattern preliminarily stored and irradiated by the pattern irradiation unit, to calculate low-resolution depth information of the target object.
(12) The image processing device according to any one of (1) to 5 and 8 to (10), wherein
the infrared image capturing unit includes two or more infrared image capturing units and
the image processing device further includes
(13) The image processing device according to (12), further including
a high-resolution depth calculation unit that calculates high-resolution depth information of the target object on the basis of the low-resolution depth information calculated by the fourth depth calculation unit and contour information of the target object extracted from the pattern-free infrared image.
(14) The image processing device according to any one of (1) to (14), wherein the pattern is a random pattern.
(15) The image processing device according to any one of (1) to (14), wherein the pattern is formed by causing an infrared ray to pass through a mask formed by electroforming.
(16) An image processing method including:
irradiating an infrared pattern onto a surface of a target object by a pattern irradiation unit; and
capturing an infrared image by an infrared image capturing unit, wherein
the pattern irradiation unit performs irradiation at a predetermined timing corresponding to an infrared image capturing unit's image capturing timing, and
the infrared image capturing unit obtains a pattern-projected infrared image in which the pattern irradiated by the pattern irradiation unit is projected on the target object, and a pattern-free infrared image in which the pattern is not projected on the target object.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2013-273231 | Dec 2013 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2014/079137 | 10/31/2014 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2015/098288 | 7/2/2015 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7103212 | Hager | Sep 2006 | B2 |
| 7430312 | Gu | Sep 2008 | B2 |
| 8797385 | Tsuboi | Aug 2014 | B2 |
| 8845107 | Coley | Sep 2014 | B1 |
| 9084002 | Tardif | Jul 2015 | B2 |
| 9098908 | Kirk | Aug 2015 | B2 |
| 9204121 | Marason | Dec 2015 | B1 |
| 9294758 | Xiong | Mar 2016 | B2 |
| 9383549 | Ueno | Jul 2016 | B2 |
| 9392262 | Dal Mutto | Jul 2016 | B2 |
| 9513768 | Zhao | Dec 2016 | B2 |
| 9538081 | Zhou | Jan 2017 | B1 |
| 9571818 | Pulli | Feb 2017 | B2 |
| 9710923 | Ohba | Jul 2017 | B2 |
| 9743065 | Kang | Aug 2017 | B2 |
| 9746318 | Sugano | Aug 2017 | B2 |
| 9774832 | Minami | Sep 2017 | B1 |
| 9894332 | Minami | Feb 2018 | B2 |
| 20050281475 | Wilson | Dec 2005 | A1 |
| 20060072914 | Arai | Apr 2006 | A1 |
| 20060291014 | Hirata | Dec 2006 | A1 |
| 20080297621 | Sun | Dec 2008 | A1 |
| 20090219253 | Izadi | Sep 2009 | A1 |
| 20100265316 | Sali | Oct 2010 | A1 |
| 20100328475 | Thomas | Dec 2010 | A1 |
| 20110025827 | Shpunt | Feb 2011 | A1 |
| 20110033088 | Rekimoto | Feb 2011 | A1 |
| 20110242334 | Wilburn | Oct 2011 | A1 |
| 20130095920 | Patiejunas | Apr 2013 | A1 |
| 20130100256 | Kirk et al. | Apr 2013 | A1 |
| 20130215235 | Russell | Aug 2013 | A1 |
| 20130229396 | Huebner | Sep 2013 | A1 |
| 20130265438 | Sugiyama | Oct 2013 | A1 |
| 20130327218 | Izzi | Dec 2013 | A1 |
| 20140002616 | Ohba | Jan 2014 | A1 |
| 20140192158 | Whyte | Jul 2014 | A1 |
| 20140241612 | Rhemann | Aug 2014 | A1 |
| 20140247364 | Kim | Sep 2014 | A1 |
| 20140307056 | Collet Romea | Oct 2014 | A1 |
| 20140307058 | Kirk | Oct 2014 | A1 |
| 20140362188 | Yokokawa | Dec 2014 | A1 |
| 20150022669 | Hall | Jan 2015 | A1 |
| 20150092018 | Kang | Apr 2015 | A1 |
| 20150229915 | Kirk | Aug 2015 | A1 |
| 20150256813 | Dal Mutto | Sep 2015 | A1 |
| 20150268450 | Ueno | Sep 2015 | A1 |
| 20150304631 | Lee | Oct 2015 | A1 |
| 20150381965 | Atanassov | Dec 2015 | A1 |
| 20160065930 | Chandra | Mar 2016 | A1 |
| 20160139674 | Watanabe | May 2016 | A1 |
| 20160194181 | Wang | Jul 2016 | A1 |
| 20160253836 | Unten | Sep 2016 | A1 |
| 20160259057 | Ito | Sep 2016 | A1 |
| 20160349043 | Lee | Dec 2016 | A1 |
| 20180234643 | Kobayashi | Aug 2018 | A1 |
| 20190033452 | Ito | Jan 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 2008-537190 | Sep 2008 | JP |
| 2009-530604 | Aug 2009 | JP |
| 2011-232150 | Nov 2011 | JP |
| 2012-103239 | May 2012 | JP |
| 2012-122975 | Jun 2012 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20170032531 A1 | Feb 2017 | US |
