SYSTEM AND METHOD OF CALIBRATING A DIRECTIONAL LIGHT SOURCE RELATIVE TO A CAMERA'S FIELD OF VIEW

Abstract

A system and method of calibrating a light source relative to an optical device comprise virtually projecting by an image projector an image onto a three-dimensional surface of at least one object, the image comprising a virtual grid comprised of predefined points onto an area having the three-dimensional surface of at least one object, each predefined point of the virtual grid corresponding to one of a plurality of termination points of beams cast by the directional light source, at least one of the termination points of the beams are on a surface of the at least one object; simultaneously capturing by at least one optical device data for a plurality of points on the virtual grid by projecting the image into a field of view of the at least one optical device; and calibrating a location and orientation of the light source relative to the optical device using the captured data for the plurality of points.

Description

FIELD OF THE INVENTION

The invention relates generally to calibration systems and methods for localizing and orienting an active directional light source relative to a camera.

BACKGROUND

Active directional light sources can provide visual indications and deliver optical energy to variable targets within their line of sight. Cameras can generally provide feedback on the location at which a directional light source is currently oriented. That location is generally referred to herein as the directional light source's terminal point. One method of calibrating the directional light source relative to the camera's field of view has the camera provide active feedback during the targeting process while the laser moves iteratively in the direction of the target. Another method calibrates the directional light source prior to targeting, which allows the light source to accurately direct the terminal point at the target in a single attempt.

The active feedback method can perform poorly in high-speed applications. This poor performance is attributable to the iterative nature of the active feedback process, wherein the terminal point of the directional light source is found repeatedly and the directional light source is then rotated in the direction of its target, with the rate of iterations being limited by the framerate of the camera and by the throughput of the visual processing algorithm used to determine the terminal point's location within an image. This method also has the disadvantage of requiring the directional light source to remain active for some period each time a new target is being acquired, even while the directional light source is not oriented towards the target. This may be unacceptable in some applications.

A directional light source that has undergone a calibration process prior to targeting is capable of being directed to a new target in a single movement, within the constraints of the calibration process it has undergone and can do so even while the directional light source is inactive. However, this method imposes a new set of constraints. For instance, current calibration methods require a known object to be within the line of sight of both the directional light source and the camera to provide a frame of reference. Often, a planar grid of known proportions with a checkerboard pattern is used. This planar grid allows a camera to determine its exact location in a three-dimensional coordinate system along with its relative orientation. The directional light source's terminal point is then guided to various points along the planar grid, either manually or using the aforementioned active feedback, and the angles for each point are recorded. From the angle data, geometrical techniques can then determine the relative location and orientation of the directional light source with respect to the camera.

Systems implementing the calibration method generally take one of two approaches. Either the planar calibration grid remains with the system as a fixture, or the planar grid is put in place only during the calibration process and is removed for normal operation of the system. Systems that retain the planar calibration grid as a fixture suffer from loss of visual range and targetable area. The accuracy of a calibration algorithm depends upon the data coming from a wide range of angles across both the camera's and the directional light source's field of view. Because of this dependence, the planar calibration grid requires a significant amount of area and blocks a large area from being both visible and targetable. Such a large planar calibration grid, which typically requires a unique textured pattern, for example, a checkerboard, is often aesthetically unappealing and can be visually confusing in some applications.

Systems that use a removable planar calibration grid often requires human intervention in order to recalibrate. Such human intervention can cause a significant decrease in the reliability of the system, as even small changes in the orientation of the camera or the mount of the directional light source can require a system recalibration. Human intervention also means that these systems cannot be deployed in unstaffed locations.

SUMMARY

All examples and features mentioned below can be combined in any technically possible way.

In one aspect, a method of calibrating a light source relative to an optical device, the method comprises the steps of virtually projecting by an image projector an image onto a three-dimensional surface of at least one object, the image comprising a virtual grid comprised of predefined points onto an area having the three-dimensional surface of at least one object, each predefined point of the virtual grid corresponding to one of a plurality of termination points of beams cast by the directional light source, at least one of the termination points of the beams are on a surface of the at least one object; simultaneously capturing by at least one optical device data for a plurality of points on the virtual grid by projecting the image into a field of view of the at least one optical device, and calibrating a location and orientation of the light source relative to the optical device using the captured data for the plurality of points.

In another aspect, a method of calibrating a light source relative to a depth camera, the method comprises the steps of: virtually projecting a virtual grid comprised of a plurality of predefined points onto a three-dimensional surface of at least one object, each predefined point of the virtual grid corresponding to one of a plurality of termination points of beams cast by the light source, at least one of the termination points of the beams are on a surface of the at least one object; capturing by the depth camera images of an area having the at least one object; causing by the light source a disturbance in one or more point clouds of the depth camera to detect the at least one of the termination points; and calibrating a location and orientation of the light source relative to the depth camera based on data of the light source associated with the images.

In another aspect, a method of calibrating at least one light source relative to at least one optical device, the method comprising the steps of: calibrating a first directional light source to a first camera and a second camera; determining a position and orientation of the first and second cameras relative to each other by processing data regarding a position and orientation of the first directional light source relative to the first and second cameras; calibrating a second directional light source to the second camera; and calculating a calibration matrix for calibrating the second directional light source to the first camera using a combination of data for calibrating the first directional light source to the first camera and the second camera and data for calibrating the second directional light source to the second camera.

In another aspect, the invention is related to a method of calibrating a movable directional light source relative to an optical device. The method comprises virtually projecting a virtual grid comprised of predefined points onto an area having at least one object disposed therein at an unknown location. The directional light source is aimed at each predefined point, in succession, of the virtual grid. The optical device captures, for each predefined point of the virtual grid at which the light source is aimed, a first image of the area while the directional light source is active and a second image of the area while the directional light source is turned off. For each predefined point of the virtual grid at which the directional light source is aimed, the first and second images captured for that predefined point are compared. A light dot is searched for in the first image captured at each predefined point based on the comparison of the first and second images captured for that predefined point. A location and orientation of the directional light source are calibrated relative to the optical device based on angle data of the directional light source associated with each image in which the light dot is found so that the directional light source can be aimed at a new target within a field of view of the optical device in a single movement while the directional light source is turned off.

In one embodiment, the virtual grid comprises a set of pan and tilt angles at which to position the directional light source to define the predefined points at which the directional light source is aimed.

The method can further comprise acquiring depth data in each image captured by the optical device, wherein the step of calibrating the location and orientation of the directional light source relative to the optical device is further based on the depth data acquired from each image in which the light dot is found.

The method can further comprise determining, when the light dot is found in a given image captured by the optical device, whether the depth data for that light dot is valid. The step of determining whether the depth data for the light dot found in the given image is valid can further comprise determining maximum and minimum depth values for pixels in a window around the light dot in the given image, and determining that the depth data for the light dot found in the given image is valid if a difference between the maximum and minimum depth values is less that a predefined threshold value.

The method can further comprise storing in memory, in response to determining that the depth data for the light dot found in the given image is valid, a data point comprised of a pixel location of the light dot in the given image, the depth data for the light dot found in the given image, pan and tilt angles of the directional light source at which the directional light source produced the light dot, and row and column data of the predefined point of the virtual grid at which the directional light source aimed when the directional light source produced the light dot; the data point is used to calibrate the location and orientation of the directional light source relative to the optical device. In addition, the method can further comprise determining that the light dot found in the given image spans a plurality of pixel locations in the given image; and averaging the spanned plurality of pixel locations to calculate the pixel location of the light dot in the given image.

In one embodiment, the method comprises applying one or more filters to the first and second images captured for each predefined point of the virtual grid at which the directional light source is aimed before comparing said first and second images. One of the filters can be a grayscale filter adapted to detect black pixels in said first and second images. One of the filters can be a linear filter weighted towards a color of light that is produced by the directional light source. One of the filters can be a linear filter weighted to an inverse of a color of light that is produced by the directional light source.

In another aspect, the invention is related to a system comprising a movable directional light source aimed, in succession, at each predefined point of a virtual grid that is projected onto an area having therein at least one object disposed at an unknown location. The system further comprises an optical device having a field of view that covers at least a portion of the area. The optical device is configured to capture, for each predefined point of the virtual grid at which the light source is aimed, a first image of the area within the field of view of the optical device while the light source is active and a second image of the area within the field of view of the optical device while the light source is turned off. A processing unit is in communication with the directional light source and the optical device. The processing unit is configured to compare the first and second images captured by the optical device for each predefined point of the virtual grid at which the light source was aimed and to search for a light dot in the first image captured for each predefined point based on the comparison of the first and second images captured for that predefined point. The processing unit is further configured to calibrate a location and orientation of the directional light source relative to the optical device based on angle data of the directional light source associated with each image in which the processing unit finds the light dot so that the directional light source can be aimed at a new target within the field of view of the optical device in a single movement while the directional light source is turned off.

In one embodiment, the virtual grid comprises a set of pan and tilt angles at which to position the directional light source to define the predefined points at which depth data are gathered.

The optical device can be further configured to acquire depth data for each captured image and wherein the calibrating of the location and orientation of the directional light source relative to the optical device is further based on the depth data acquired from each image in which the processing unit finds the light dot.

The processing unit may be further configured to determine, when the light dot is found in a given image captured by the optical device, whether the depth data for the light dot is valid. In one embodiment, the processing unit may be further configured to determine maximum and minimum depth values for pixels in a window around the light dot in the given image, and determine that the depth data for the light dot found in the given image is valid if a difference between the maximum and minimum depth values is less that a predefined threshold value. In addition, the processing unit may be further configured to store in memory, in response to determining that the depth data for the light dot found in the given image is valid, a data point comprised of a pixel location of the light dot in the given image, the depth data for the light dot found in the given image, pan and tilt angles of the directional light source when the directional light source produced the light dot, and row and column data of the predefined point of the virtual grid at which the directional light source aimed when the directional light source produced the light dot. The data point is used by the processing unit to calibrate the location and orientation of the directional light source relative to the optical device.

The processing unit may be further configured to determine that the light dot found in the given image spans a plurality of pixel locations in the given image; and to average the spanned plurality of pixel locations to calculate the pixel location of the light dot in the given image.

The processing unit may be further configured to apply one or more filters to the first and second images captured for each predefined point of the virtual grid when comparing said first and second images. One of the filters may a grayscale filter adapted to detect black pixels in said first and second images, a linear filter weighted towards a color of light that is produced by the directional light source, or a linear filter weighted to an inverse of a color of light that is produced by the directional light source.

The optical device may comprise a camera having a depth sensor.

In yet another aspect, the invention relates to a method for calibrating location and orientation of a directional light source relative to a field of view of an optical device. The method comprises directing light from the directional light source to a plurality of points on a virtual grid configured to overlay the field of view of the optical device: capturing, by the optical device, for each point of the plurality of points on the virtual grid to which the directional light source directs the light, an image; locating a light dot in a plurality of captured images; and calibrating the location and orientation of the directional light source relative to the field of view of the optical device based on coordinates of each located light dot and on relative coordinates of the optical device.

The method may further comprise acquiring depth data in each image captured by the optical device, and wherein calibrating the location and orientation of the directional light source relative to the field of view of the optical device is further based on the depth data acquired from each captured image in which the light dot is located.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and further advantages of this invention may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like numerals indicate like structural elements and features in various figures. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Various aspects of at least one embodiment are discussed below with reference to the accompanying figures, which are not intended to be drawn to scale. For purposes of clarity, not every component may be labeled in every figure. In the figures:

FIG. 1 is a diagram of an embodiment of a computer-vision system with light-guidance capability operating within an environment containing one or more unknown objects;

FIG. 2 is a diagram of an example of a virtual grid projected onto the objects in the environment shown in FIG. 1;

FIG. 3A and FIG. 3B together are a flowchart of an example of a calibration process used by the system of FIG. 1;

FIG. 4A and FIG. 4B together are a process flow diagram showing an embodiment of a method used to find laser dots used by the system of FIG. 1;

FIG. 5 is a flowchart of an example of a process used to determine if the depth measurement for a given data point can be considered valid.

FIG. 6 is a flowchart of an example of an argument minimization process used to find a set of location and orientation parameters that best fits the data collected by the camera of light dots corresponding to where the directional light source pointed when rotated to various angles.

FIG. 7 is a diagram of a multi-camera, multi-sensor computer-vision system operating in a room, in accordance with some embodiments.

DETAILED DESCRIPTION

Described herein are computer-vision systems with light-guidance capability and methods of calibrating a directional light source relative to the field of view of the camera. The systems can determine the relative position and orientation of the directional light source to camera with enough precision to enable the directional light source to be aimed at targets within the field of view of the camera without using any known object or specialized visual markers. The calibration methods described herein provides for targeting with speed and accuracy and allow the light source to be activated only when aimed directly at the target during normal light-guidance operation. Accuracy, in general, is measured in degrees offset from a target feature. Targeting is more accurate than active feedback because it uses a large dataset, which allows slight random errors to cancel out and ignores areas where data are difficult to detect during calibration, but can extrapolate data from elsewhere to determine accurate targeting information in those areas.

In one embodiment, a computer-vision system achieves this targeting capability using a camera with an infrared channel by which the camera can determine the approximate distance to locations within its field of view. Using this distance-measurement capability, the computer-vision system finds the location of the terminal point of the directional light source in a three-dimensional coordinate system. The additional depth data relaxes the requirements for data collection, allowing data collection from a set of data points across the field of view without being constrained to a single plane or other specialized surface.

To complete the calibration process, an argument minimization algorithm (e.g., Levenberg-Marquardt) operates on a model of the directional light source to find a set of location and orientation parameters that best fits the data, collected by the camera, of the terminal point locations corresponding to when the directional light source is rotated to various angles.

FIG. 1 shows an embodiment of a computer-vision system 100 with light-guidance capability including a camera 102, a directional light source 104, and a computer system 106 (or processing unit). The camera 102 is connected to the computer system 106 by a communication cable 108; the directional light source 104 is connected to the computer system 106 by a communication cable 110. In one embodiment, the communication cables 108, 110 are bidirectional serial communication cables.

In some embodiments, the camera 102 is an RGB (red-green-blue) camera equipped with a sensor through which the camera acquires color images of persons, objects (or things) within its field of view 112. The camera 102 may also include a depth sensor for measuring distance to those objects or persons within the field of view 112. Such a depth sensor captures depth data for each pixel in a captured image. The camera 102 may be, for example, an Intel® Realsense™ camera produced by Intel Corporation of Santa Clara, Calif. In another embodiment, the depth sensor is a separate device from the camera. Whether or not the depth sensor is included in the camera, a mapping may be performed, by the camera 102 or the computer system 106, to calibrate the fields of view of the depth sensor and the camera relative to each other so that a measured distance to an object or person is with respect to the camera's field of view 112.

In another embodiment, the system does not include an RGB camera. Here, directional light can cause a disturbance with respect to the point clouds formed by a depth camera, noting that many depth camera's point clouds are disturbed by high intensity light. This disturbance can be used to detect the “terminal point” of the directional light source rather than the feedback of an RGB camera, which allows for a simpler more cost effective system by removing the need for an RGB camera.

In one embodiment, the directional light source 104 is comprised of a 5-mW laser seated in a two-axes gimbal with a rigid mount. The laser gimbal is stationed in the vicinity of the camera 102 such that the terminal point of the laser may be directed into the camera's field of view. Other embodiments of directional light sources include projectors and flashlights.

The computer system 106 includes a processor, data store (i.e., memory), I/O interfaces, and program code (not shown) that, when executed, performs the methods of calibration to determine relative position and orientation of the camera 102 and light source 104, as described herein.

During a calibration process, the directional light source 104 casts a set of beams as the gimbal iteratively turns, starting with an initial beam 114, followed by beam 116, then beam 118, and then beam 120. An angle of each turn is chosen with an objective of getting enough datapoints and a low enough final average error between the test data and the calculated output, as compared to a threshold setting that is determined based on the application (i.e., use) for which the system is being calibrated. In general, smaller angles between datapoints provide more opportunities to gather data for the same range but increase the time taken by the calibration process to operate. In one embodiment, an angle of 5 degrees is used for each turn of the gimbal. In addition, the calibration process does not require any object in the foreground to operate. Each beam falls upon a wall in the background or upon any object of unknown (i.e., unconstrained) geometry which may be present in front of the wall at the time of targeting. In the present example, the termination points 130, 134 of beams 114, 116, respectively, are on a surface of object 122, the termination point 138 of beam 118 is on object 124, and the termination point 142 of beam 120 is on object 126. Also, each termination point of the beams 114, 116, 118, and 120 falls within the camera's field of view 112, the boundaries of which are represented by lines 128. The camera 102 is used to detect each termination point and determines the depth (i.e., distance) to that termination point. For example, the camera is used to detect the termination point 130 of beam 114 along path 132, the termination point 134 of beam 116 along path 136, the termination point 138 of beam 118 along path 140, and the termination point 142 of beam 120 along path 144.

FIG. 2 shows an example embodiment of a virtual grid 200 that is virtually projected onto the potential background objects 122, 124, 126 and onto the floor 202 and walls 204 of the enclosing room 206 during the calibration process. The virtual grid 200 is not visible; rather the virtual grid represents the set of all points to which the processing unit 106 drives the directional light source during the calibration process. In one embodiment, the virtual grid 200 comprises a set of pan and tilt angles at which to position the laser gimbal to establish the test points at which data are gathered. This set of pan and tilt angles can encompass the entire angular range of the laser gimbal 104. In one embodiment, an approximation of the relative position and orientation of the laser gimbal 104 suffices to generate a set of test points that fall within the field of view 112 of the camera 102. Information, such as the camera and laser are physically close to each other and point generally in the same direction, serves to formulate an approximation that avoids test points that are outside of the field of view of the camera. The approximation can thus help speed up the calibration process by avoiding test points the camera cannot see. Accordingly, in principle, the approximation of the relative position and orientation of the laser gimbal helps determine test points that are used to determine more precise values of the relative position and orientation.

The virtual grid 200 comprises a set of column lines (running vertically) and row lines (running horizontally). Because the pan and tilt motors that aim the laser of the laser gimbal operate in a spherical coordinate system, the virtual grid 200 appears warped and column and row lines appear as parabolas when projected onto a plane. Points on the virtual grid 200 occur where a column intersects a row. Each point on the virtual grid 200 corresponds to one of the termination points of the beams cast by the directional light source 104. Depth data are collected at each point. The example virtual grid 200 shown in FIG. 2 has five rows, five columns, and twenty-five test points (also referred to as data collection points). Other embodiments can have fewer or more rows and/or columns, and thus fewer or more data collection points.

For purposes of illustrating an example calibration process, a first test point 208 of the virtual grid 200 corresponds to the upper right corner (where the rightmost column intersects the topmost row), and a second point 210 of the virtual grid 200 lies immediately below the first point 210 (where the rightmost column intersects the second row). Successive points 212, 214, and 216 fall on the rightmost column at the intersections of the third, fourth, and fifth rows, respectively. The points 208, 210, 212, 214, and 216 are considered the “first pass” made by the iteratively moved directional light source. In this embodiment, the calibration process collects data in a set of columns, traveling down one column from its top test point to its bottom test point, before moving to an adjacent column (if any) and repeating the data collection for that column starting with its top test point. One column of data collection attempts from top to bottom is considered “a pass”.

The next pass begins at the top of the second column from the right, at the point 218, and successively progresses down the column to the bottommost point 220. The pattern repeats with the next column to the left, with points on the bottom row being followed by the point on the top row of the next column to the left, until the final point 222 at the intersection of the leftmost column with the bottom row is reached.

In embodiments where the directional light source includes at least one projector for mapping projections of an image onto 3D surfaces, the simultaneous capture of data on many points on the virtual grid may be performed by projecting a more complex image into the camera's field of view. In a projected image, the potential for higher accuracy of point location measurement occurs when using the corners of a checkerboard projection.

FIG. 3A and FIG. 3B together show an embodiment of a calibration process 300, which collects data used to calibrate the laser gimbal with respect to the camera. The calibration process can be considered to have three major processes: a data collection process, which uses camera data and laser control as input data and produces a set of data points, including pixel location (u, v), depth (s), and laser angles (x_tilt, y_pan) as output data; a data preprocessing process, which uses pixel location (u, v), depth (s), and intrinsic parameters of the camera as input data and produces the absolute positions of the laser dots (X,Y, Z) as output data; and an argument minimization stage, which uses the absolute position of the laser dots (X,Y, Z) and corresponding laser angles (x_tilt, y_pan) as input data and produces the rotational and translational parameters (R, t) as output data.

In the description of the calibration process, reference is made to elements of FIG. 1 and FIG. 2. As shown in FIG. 3A, the laser gimbal 104 is initialized at step 302. Initialization entails powering up and homing the pan and tilt axis motors of the laser gimbal to a repeatable coordinate system. At step 304, a virtual grid, such as the virtual grid 200 described in connection with FIG. 2, is generated.

Generation of the virtual grid (i.e., the pan and tilt angles) can be predetermined. In one embodiment, the virtual grid is generated using a set of initial parameters including pan range, tilt range, center position, number of passes, and the number of test points per pass. Points per pass can be increased to maximize accuracy, at the cost of adding more time to the data collection step. Some of those inputs can be determined from an approximation of the relative position and orientation of the gimbal, as previously described, to keep all collected data points within the field of view. For situations where an estimate is not available, large ranges are recommended to maximize the likelihood that points fall within the field of view. Other factors for generating the virtual grid include, but are not limited to, the amount of time within which to perform calibration and the desired accuracy. For situations where an approximation of the relative orientation and position of the directional light source to the camera are known, the center position, pan range, and tilt range can be configured.

The laser (i.e., the directional light source) is turned off (step 306) as data collection begins in order to allow for images of the background to be taken. The laser is then driven (step 308) to the first test point on the virtual grid 200. The first test point is typically a point at one of the corners of the virtual grid 200. While the laser remains off, an image of the area within the camera's field of view is captured and saved (step 310). The laser is then turned back on (step 312) and another image of the area within the camera's field of view is captured (step 314). This second captured image potentially contains a laser dot (i.e., an illuminated termination point of the directional light source). The laser is then returned to the off state (step 316) to prepare for the next test point.

As shown in FIG. 3B, the two images are compared (step 318) by the processing unit to determine the presence and location of a laser dot. If, at step 320, a laser dot is found, a depth validation process is run (step 322) to ensure that the depth value of the laser dot's location captured by the camera was determined accurately. If, at step 324, the depth validation process determines the depth value was valid, the valid data point is saved (step 326), together with the pixel location (x, y) of the laser dot, the depth value, and the pan and tilt angles of the laser gimbal. The calibration process resumes at step 308 (FIG. 3A) to either move the laser gimbal to the next point on the virtual grid (if data collection at all virtual grid test points has not yet been performed) and then repeat the pattern of data collection at that point, or to finish the calibration process (after data collection at all points has been performed). Further, if a laser dot is not found at step 320, or if a laser dot is found but the depth value captured by the camera could not be validated at step 322, the calibration process returns to step 308 of FIG. 3A.

When the calibration process resumes at step 308 (FIG. 3A), and if all virtual grid test points have been examined, the calibration process 300 determines (step 328) whether enough valid datapoints have been collected. In one embodiment, six valid datapoints constitute enough datapoint collection, which equals the number of arguments to be minimized to determine the relative position and orientation. More than six valid datapoints is preferred. Upon completion of successful data collection, the data set is composed of a set of pan and tilt values for the laser gimbal, along with the corresponding row, column, and depth values describing where the laser dot was found within the camera's field of view. An argument minimization process runs (step 330) if enough valid datapoints have been found; otherwise the calibration process 300 is deemed to have failed (step 332). In the event the calibration process 300 is deemed to have failed, it may be restarted. In cases where failure occurs repeatedly, adjustments to the virtual grid can be considered, including increasing the pan or tilt range, and increasing the number of passes.

FIG. 4A and FIG. 4B together show an embodiment of a process 400 for searching for a laser dot in an image. This laser dot search process 400 can be used in step 318 of the previously described calibration process 300. The camera captures (step 310 of FIG. 3A) a first image 402 of a scene within the camera's field of view while the laser is turned off and captures (step 314 of FIG. 3A) a second image 404 of the scene while the laser is turned on. A set of filters are applied to both images 402, 404.

One filter is a simple grayscale filter, in which the red, green, and blue values are averaged, to produce a grayscale version 406, 408 of each input image 402, 404, respectively. These greyscale images 406, 408 simplify the detection of pitch-black pixels, which are a common form of noise for a camera setup. A threshold value 410, 412 of 0 is then used to generate a bitmask 411, 413 for each greyscale image 406, 408, respectively, that represents the locations of all non-black pixels in those greyscale images. As used herein, a bitmask has one bit for each pixel for a given image. A value of 0 represents a black pixel, and a bitmask with a threshold of 0 provides an output which represents all non-black pixels.

A second filter is a linear weighted filter used to find the characteristic color of the laser in each input image 402, 404. The weightings for the RGB values are biased towards the color of the laser, making the color of the laser stand out more in the image, and filters out white, thereby producing two target-colored filtered images 414, 416 for the input images 402, 404, respectively. The absolute difference 418 of the two target-colored filtered images 414, 416 is determined, thereby producing a single target-color filtered image 420. This target-color filtered image 420 tends to highlight the areas of the image illuminated by the laser, but also tends to include reflections of the laser dot, which may make it difficult to determine the laser location directly from this image 420. This target-color filtered image 420 can also include considerable noise caused by the existence of pixels that appear black in one of the input images 402, 404 and not in the other. An adaptive target-color threshold 422 is then used to generate a bitmask 423 representing the locations of particularly red pixels in the target-color filtered image 420. The threshold 422 is designed to identify the pixels that have become redder as a result of the laser being turned on, which is represented by high-value pixels in a red-filtered image of the difference 418 of the two images 414, 416.

A third filter applied to the input images 402, 404 is a linear weighted filter that produces two inverted color filtered images 424, 426, respectively, that are biased towards white, but invariant to the color of the laser. This inverted color filtering is accomplished by weighting the linear filter towards the inverse of the laser's color. The absolute difference 428 of the two inverted color filtered images 424, 426 is determined, which produces a single filtered inverted-color image 430. This filtered inverted-color image 430 is useful because the image 430 retains a white dot at the center of the primary laser dot, while ignoring reflections which tend not to have a central white dot. An adaptive inverted color threshold 432 is then used to generate a bitmask 433 representing the locations of the white and inverted color pixels in the filtered inverted-color image 430.

A logical AND function 434 is then used to find the overlap, also known as the intersection, of the four bitmasks 411, 413, 423, and 433. The logical AND function 434 produces a bitmask 435 with a 1 value for each pixel where all four bitmasks 411, 413, 423, and 433 have a 1 value (i.e., they overlap), and a 0 value for every other pixel that does not meet this logical AND criteria. Referring to FIG. 4B, the resulting overlap bitmask 435 passes to an erosion filter 436. This erosion filter 436 eliminates any small areas of pixels (e.g., one to six) that may have made it through the previous filters, which likely represent noise, and produces a final bitmask 437. If the original image 404, taken while the laser was on, did not contain a laser dot, the final bitmask 437 will be black. If the final bitmask 437 is black (block 438), a laser dot is deemed to not have been found (block 440). If any pixels remain that are not black (block 438), then the average location (i.e., centroid) of these pixels are found (block 442), and this average location is deemed the location of the laser dot (block 444).

FIG. 5 shows an embodiment of a depth validation process 500 that may be used at step 322 of the calibration process 300 described in connection with FIG. 3A and FIG. 3B. At step 502, depth values are acquired from a window centered around the position of the laser dot found in block 444 of FIG. 4B and evaluated. The size of the window can vary depending on the kind of environment of the camera and directional light source. For example, an 8-pixel by 8-pixel window is an effective size for an indoor environment. At step 504, the maximum and minimum values of the measured depths are found and the difference between these values are determined. The difference is compared (step 506) against a threshold to determine if the depth value at the exact location of the laser dot can be considered valid. If the difference is less than the threshold, the depth value is deemed valid (step 508); if the difference is greater than the threshold, the depth value is deemed invalid (step 510). This depth validation process 500 can screen out depth values that are measured on surfaces substantially perpendicular to the camera's field of view, that are near the edges of objects, or that are in areas with high noise levels. These circumstances contribute to high error levels in depth measurements (and thus lead to greater differences between measured minimum and maximum values) and screening out datapoints that exceed the threshold increases the quality of the final calibration.

FIG. 6 shows an embodiment of an argument minimization process 600 that may be used at step 330 of the calibration process 300 described in connection with FIG. 3A and FIG. 3B. In general, the argument minimization process 600 finds a set of location and orientation parameters that best fits the data collected, by the camera, of light dots that correspond to where the directional light source pointed when rotated to various angles.

The following equation is used to describe (step 602) the mapping from absolute cartesian coordinates (X, Y, Z) representing the location of a laser dot relative to the laser gimbal to the pixel row and column coordinates (u, v), with the depth value as a scaling factor (s), of that laser dot found in an image.

$\begin{array}{cc}s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left(R|t\right)\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]& \mathrm{Eq}.1\end{array}$

In EQ. 1, the A matrix represents the intrinsic parameters, such as optical center and focal length, of the camera, which may be known for a given camera, or which may be pre-calibrated once with the use of a checkerboard pattern as the intrinsic parameters do not change over time or with the placement of the camera. The R matrix and t vector represent the rotational and translational transformation between the laser gimbal's and the camera's coordinate systems, respectively. An objective of the argument minimization process 600 is to solve for the R matrix, which represents the orientation (i.e., rotational) difference, and the t vector, which represents the position (i.e., translational) difference. The R matrix and the t vector are determined from the dataset of pixel row and column coordinates (u, v), depth values (s), and laser gimbal angle coordinates (pan, tilt) using Levenberg-Marquardt optimization.

As the absolute coordinates of the laser dot are unknown, and only the pan and tilt angles of the laser gimbal are known, the following relationships are defined (step 604) between the pan and tilt angles, y_pan and x_tilt respectively, and the absolute cartesian coordinates of the laser dot, X Y and Z. Y_pan and X_tilt are angular coordinates representing the current state of the laser gimbal.

$\begin{array}{cc}\tan \left(y_{pan}\right)=\frac{X}{Z}& \mathrm{Eq}.2\\ \tan \left(x_{tilt}\right)=-\frac{Y}{\sqrt{\left(X*X\right)+\left(Z*Z\right)}}& \mathrm{Eq}.3\end{array}$

From the above relationships, the following equations for the X and Y values can be derived.

$\begin{array}{cc}X=\tan \left(y_{pan}\right)*Z& \mathrm{Eq}.4\\ Y=-\tan \left(x_{\mathrm{tilt}}\right)*Z*\sqrt{\tan \left(y_{pan}\right)*\tan \left(y_{pan}\right)+1}=-\frac{\tan \left(x_{\mathrm{tilt}}\right)}{\cos \left(y_{pan}\right)}*Z& \mathrm{Eq}.5\end{array}$

Using the above equations (Eq. 4 and Eq. 5) for the X and Y positions, a system of equations (Eq. 6) can be constructed using the transformation from the laser gimbal's coordinate system to the camera's coordinate system along with the following equations (Eq. 7 and Eq. 8) corresponding to the respective ratios between the X and Z coordinates and the Y and Z coordinates of the position of the laser dot that are solely dependent on the pan and tilt angles of the laser gimbal.

$\begin{array}{cc}\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=R^{-1}\left(A^{-1}s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]\right)+t& \mathrm{Eq}.6\\ \tan \left(y_{pan}\right)=\frac{X}{Z}& \mathrm{Eq}.7\\ \frac{\tan \left(x_{\mathrm{tilt}}\right)}{\cos \left(y_{pan}\right)}=\frac{Y}{Z}& \mathrm{Eq}.8\end{array}$

In addition to the above relationships, a metric for error is defined. (step 606)

The Levenberg-Marquardt algorithm is then used (step 608) along with the pixel row and column coordinates (u, v), depth values (s), and laser gimbal angle coordinates (pan, tilt) collected in the steps described in FIG. 3A and FIG. 3B to find parameter values for the rotational and translational transformations that minimize the error of the above equations for transforming the collected data from the camera's row, column, and depth values to the laser gimbal's pan and tilt angles. These rotational and translational parameter values serve to localize and orient the camera relative to the laser gimbal's coordinate system.

After completing the above calibration process, the appropriate pan and tilt angles to direct the laser gimbal towards any X, Y, and Z absolute coordinates can be calculated (step 610) directly using equations 9 and 10 below. These equations are algebraic manipulations of equations 7 and 8.

$\begin{array}{cc}{y}_{pan}=\mathrm{atan}\left(\frac{X}{Z}\right)& \mathrm{Eq}.9\\ x_{\mathrm{tilt}}=\mathrm{atan}\left(\frac{Y}{Z}\cos \left(y_{pan}\right)\right)& \mathrm{Eq}.10\end{array}$

The X, Y and Z values required for the above calculation can be calculated from camera data using equation 6, using the rotational and translational parameter values for R and t respectively, that were determined during the argument minimization process.

Some or all of one or more methods described herein may include the use of the same directional light source in the same position, orientation, location, or the like with respect to multiple different cameras. Here the relative position, orientation, and/or location-related information of the cameras can be calculated relative to each other. For example, if a single directional light source is calibrated to two cameras, then the relative position and orientation of the first camera relative to the second camera can be calculated. This can be chained together so that if the light source is calibrated to the two cameras as described above, and a second light source is also calibrated to the second camera, an approximate calibration matrix can be calculated for the second light source and first camera using the data captured by the other calibration data exchanges. Therefore, applications including room 700 in FIG. 7 designated for holding objects and in which three cameras 702A, 702B, 702C (generally, 702) and two light sources 704A, 704B (generally, 704) and/or other computer-vision-based object tracking apparatuses are arranged for identifying and/or tracking the objects, which may include but not be limited to, packages, parcels, boxes, equipment, tools, food products, bottles, jars, cans, or even people. Although three cameras and two light sources are shown and described, the embodiments may include more than three cameras and two light sources. Here, the relative locations and orientations of a plurality of various computer-vision-based object tracking apparatuses, e.g., combinations of light sources 704 and depth cameras 702 in the room 700, may be calibrated after which a point cloud representing objects in the room 700 can be formed and stitched together from the calibration data of the location relationships of the apparatuses. For example, relative locations and orientations of the cameras are calibrated to allow a stitching together of a single point cloud in a location, e.g., the room 700, where the cameras coexist with the directional light sources.

As will be appreciated by one skilled in the art, aspects of the systems described herein may be embodied as a system, method, and computer program product. Thus, aspects of the systems described herein may be embodied in entirely hardware, in entirely software (including, but not limited to, firmware, program code, resident software, microcode), or in a combination of hardware and software. All such embodiments may generally be referred to herein as a circuit, a module, or a system. In addition, aspects of the systems described herein may be in the form of a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may be a non-transitory computer readable storage medium, examples of which include, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof.

As used herein, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, device, computer, computing system, computer system, or any programmable machine or device that inputs, processes, and outputs instructions, commands, or data. A non-exhaustive list of specific examples of a computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), a USB flash drive, an non-volatile RAM (NVRAM or NOVRAM), an erasable programmable read-only memory (EPROM or Flash memory), a flash memory card, an electrically erasable programmable read-only memory (EEPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), a DVD-ROM, an optical storage device, a magnetic storage device, or any suitable combination thereof.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. As used herein, a computer readable storage medium is not a computer readable propagating signal medium or a propagated signal.

Program code may be embodied as computer-readable instructions stored on or in a computer readable storage medium as, for example, source code, object code, interpretive code, executable code, or combinations thereof. Any standard or proprietary, programming or interpretive language can be used to produce the computer-executable instructions. Examples of such languages include Python, C, C++, Pascal, JAVA, BASIC, Smalltalk, Visual Basic, and Visual C++.

Transmission of program code embodied on a computer readable medium can occur using any appropriate medium including, but not limited to, wireless, wired, optical fiber cable, radio frequency (RF), or any suitable combination thereof.

The program code may execute entirely on a user's device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computer or entirely on a remote computer or server. Any such remote computer may be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Additionally, the methods described herein can be implemented on a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit element(s), an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit such as discrete element circuit, a programmable logic device such as PLD, PLA, FPGA, PAL, or the like. In general, any device capable of implementing a state machine that is in turn capable of implementing the proposed methods herein can be used to implement the principles described herein.

Furthermore, the disclosed methods may be readily implemented in software using object or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system may be implemented partially or fully in hardware using standard logic circuits or a VLSI design. Whether software or hardware is used to implement the systems in accordance with the principles described herein is dependent on the speed and/or efficiency requirements of the system, the particular function, and the particular software or hardware systems or microprocessor or microcomputer systems being utilized. The methods illustrated herein however can be readily implemented in hardware and/or software using any known or later developed systems or structures, devices and/or software by those of ordinary skill in the applicable art from the functional description provided herein and with a general basic knowledge of the computer and image processing arts.

Moreover, the disclosed methods may be readily implemented in software executed on programmed general-purpose computer, a special purpose computer, a microprocessor, or the like. In these instances, the systems and methods of the principles described herein may be implemented as program embedded on personal computer such as JAVA® or CGI script, as a resource residing on a server or graphics workstation, as a plug-in, or the like. The system may also be implemented by physically incorporating the system and method into a software and/or hardware system.

While the aforementioned principles have been described in conjunction with a number of embodiments, it is evident that many alternatives, modifications, and variations would be or are apparent to those of ordinary skill in the applicable arts. References to “one embodiment”, “an embodiment”, “another embodiment”, or the like are not necessarily mutually exclusive and mean that a specific feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment described herein. A reference to a specific embodiment within the specification do not necessarily all refer to the same embodiment. The features illustrated or described in connection with one exemplary embodiment may be combined with the features of other embodiments in any manner consistent with at least one of the principles disclosed herein. Accordingly, it is intended to embrace all such alternatives, modifications, equivalents, and variations that are within the spirit and scope of the principles described herein.

Claims

1. A method of calibrating a light source relative to an optical device, the method comprising the steps of: virtually projecting by an image projector an image onto a three-dimensional surface of at least one object, the image comprising a virtual grid comprised of predefined points onto an area having the three-dimensional surface of at least one object, each predefined point of the virtual grid corresponding to one of a plurality of termination points of beams cast by the directional light source, at least one of the termination points of the beams are on a surface of the at least one object;simultaneously capturing by at least one optical device data for a plurality of points on the virtual grid by projecting the image into a field of view of the at least one optical device; andcalibrating a location and orientation of the light source relative to the optical device using the captured data for the plurality of points.

2. The method of claim 1, further comprising the step of storing in memory, in response to determining that data in the given image is valid, a point of the plurality of points comprised of a pixel location in the given image, the data for the point found in the given image, pan and tilt angles of the light source at which the light source produced the virtual image, and row and column data of the predefined point of the virtual grid at which the directional light source aimed when the light source produced the virtual grid, the data point being used to calibrate the location and orientation of the light source relative to the optical device.

3. The method of claim 2, further comprising the steps of: determining that the point found in the given image spans a plurality of pixel locations in the given image; andaveraging the spanned plurality of pixel locations to calculate the pixel location of the point in the given image.

4. The method of claim 1, wherein the step of applying one or more filters to first and second images captured for each predefined point of the virtual grid at which the light source is aimed before comparing said first and second images.

5. The method of claim 1, wherein the image projector is a directional light source that maps projections onto the three-dimensional surface.

6. The method of claim 1 wherein the projected image includes a checkerboard projection having corners that include the predefined points of the virtual grid, wherein the method further includes performing a point location measurement process on the corners of the checkerboard projection.

7. A method of calibrating a light source relative to a depth camera, the method comprising the steps of: virtually projecting a virtual grid comprised of a plurality of predefined points onto a three-dimensional surface of at least one object, each predefined point of the virtual grid corresponding to one of a plurality of termination points of beams cast by the light source, at least one of the termination points of the beams are on a surface of the at least one object;capturing by the depth camera images of an area having the at least one object;causing by the light source a disturbance in one or more point clouds of the depth camera to detect the at least one of the termination points; andcalibrating a location and orientation of the light source relative to the depth camera based on data of the light source associated with the images.

8. The method of claim 7, wherein the virtual grid comprises a set of pan and tilt angles at which to position the light source to define the predefined points at which the light source is aimed.

9. The method of claim 7, further comprising the step of acquiring depth data in each image captured by the optical device, and wherein the step of calibrating the location and orientation of the directional light source relative to the optical device is further based on the depth data acquired from each image in which the light dot is found.

10. The method of claim 9, further comprising the step of storing in memory, in response to determining that the depth data for the light dot found in the given image is valid, a data point comprised of a pixel location of the light dot in the given image, the depth data for the light dot found in the given image, pan and tilt angles of the directional light source at which the directional light source produced the light dot, and row and column data of the predefined point of the virtual grid at which the directional light source aimed when the directional light source produced the light dot, the data point being used to calibrate the location and orientation of the directional light source relative to the optical device.

11. The method of claim 6, further comprising the steps of: determining that the light dot found in the given image spans a plurality of pixel locations in the given image; andaveraging the spanned plurality of pixel locations to calculate the pixel location of the light dot in the given image.

12. A method of calibrating at least one light source relative to at least one optical device, the method comprising the steps of: calibrating a first directional light source to a first camera and a second camera;determining a position and orientation of the first and second cameras relative to each other by processing data regarding a position and orientation of the first directional light source relative to the first and second cameras;calibrating a second directional light source to the second camera; andcalculating a calibration matrix for calibrating the second directional light source to the first camera using a combination of data for calibrating the first directional light source to the first camera and the second camera and data for calibrating the second directional light source to the second camera.

13. The method of claim 12, wherein the relative locations and orientations of the first and second cameras are calibrated to allow a stitching together of a single point cloud in a location where the first and second cameras coexist with the first and second directional light sources.