Agile depth sensing using triangulation light curtains

Information

  • Patent Grant
  • 11972586
  • Patent Number
    11,972,586
  • Date Filed
    Wednesday, September 25, 2019
    4 years ago
  • Date Issued
    Tuesday, April 30, 2024
    15 days ago
Abstract
A method to dynamically and adaptively sample the depths of a scene using the principle of triangulation light curtains is described. The approach directly detects the presence or absence of obstacles (or scene points) at specified 3D lines in a scene by sampling the scene. The scene can be sampled sparsely, non-uniformly, or densely at specified regions. The depth sampling can be varied in real-time, enabling quick object discovery or detailed exploration of areas of interest. Once an object is discovered in the scene, adaptive light curtains comprising dense sampling of a region of the scene containing the object, can be used to better define the position, shape and size of the discovered object.
Description
BACKGROUND OF THE INVENTION

3D sensors play an important role in the deployment of many autonomous systems, including field robots and self-driving cars. However, there are many tasks for which it may not be necessary to use a full-blown 3D scanner. As an example, a self-guided vehicle on a road or a robot in the field does not need a full-blown 3D depth sensor to detect potential collisions or monitor its blind spot. Instead, what is necessary is for the vehicle to be able to detect if any object comes within a pre-defined perimeter of the vehicle to allow for collision avoidance. This is a much easier task than full depth scanning and object identification.


Depth sensors like LIDARs and the Microsoft Kinect® use a fixed depth acquisition strategy that is independent of the scene of interest. Due to the low spatial and temporal resolution of these sensors, this strategy can undersample parts of the scene that are important, for example, small and/or fast-moving objects, or oversample areas that are not informative for the task at hand, for example, a fixed planar wall.


A light curtain is a safety device that detects nearby obstacles (e.g, a human) to stop the operation of a machine. They can be used, for example, in garage doors and/or elevators to stop the doors from closing when an extended hand blocks them or on factory floors around dangerous machinery. Light curtains operate on a simple principle that an obstacle is detected when it obstructs the line of sight between a source and a sensor. Even though they are simple, light curtain systems must be specifically customized for each machine and task, hindering their broad use for vision and robotics.


Recently, this principle has been extended to generate light curtains of any shape (i.e., a ruled surface) using a line sensor and a line source. Here, an obstacle is detected when it intersects both the plane of illumination generated from the line light source and the imaging plane captured by the line sensor. In some examples, the illumination and imaging planes are rotated using steerable mirrors at different velocities to sweep out an arbitrary ruled curtain surface in 3D. Such triangulating light curtains can be highly flexible and can be useful for obstacle detection and avoidance for autonomous navigation. Such light curtains are described in related U.S. patent application Ser. No. 16/470,885, entitled “Programmable Light Curtains”, of which this application is a continuation-in-part.


SUMMARY OF THE INVENTION

A method and system to dynamically and adaptively sample the depths of a scene using the principle of triangulation light curtains is disclosed herein. The method directly detects the presence or absence of obstacles (or scene points) at specified 3D lines. These 3D lines can be sampled sparsely, non-uniformly, or densely at specified regions. The depth sampling can be varied in real-time, enabling quick object discovery or detailed exploration of areas of interest.


These results are achieved using a novel prototype light curtain system that is based on a 2D rolling shutter camera with higher light efficiency, working range, and faster adaptation than prior art works, making it useful broadly for autonomous navigation and exploration.


A general framework for agile depth sensing for vision and robotics is developed herein. In some examples, the triangulation light curtain intersects the scene along 3D lines. Thus, instead of capturing 3D data of the scene in the entire volume of interest (using say, a LIDAR or Kinect®, this framework allows the flexible sampling of the depths along 3D lines in a scene over time. The depth sampling detects the presence or absence of obstacles (or scene points) at these locations in real-time without additional computation.


The depth sampling could be sparse, non-uniform (including random), or dense at specified 3D surfaces. The sparse sampling can be used to adaptively increase the spatial density of the depths in the regions of interest as specified by an application. In some embodiments, the sparse sampling can be used to adaptively increase the spatial density of the depths in only the regions of interest. Alternatively, objects in the scene can be discovered quickly by initial random sampling followed by adaptive sampling of depths. The depth sampling can be rapidly varied over time, depending on the task at hand.


This sensing framework has several advantages over depth sensing that uses a fixed acquisition strategy independent of the scene. First, it is possible to capture small, thin, and fast-moving objects that are typically difficult for low frequency uniform angular resolution LIDAR, or low spatial resolution Kinect®-like sensors to detect. Examples of such objects include thin wires, meshes or balls thrown at high speed.


The fast and dense 3D capture of objects (e.g. pedestrians) far away from the sensor is possible with the system of the invention, even when the objects are barely visible in a LIDAR point cloud, allowing for better detection, recognition and tracking of such objects. The framework also allows a robot to explore a region of interest based on initial sparse depth estimates. By continuously and sparsely sampling the scene as the robot moves, it is possible to simultaneously detect obstacles and map the 3D scene.


To achieve these results, a novel design for triangulation light curtains that uses a 2D camera and a laser line is disclosed. The rolling shutter of the 2D camera and the rotating laser line triangulate at a set of 3D lines in the scene forming a light curtain. By controlling the pixel clock and steering the source mirror, it is possible to generate arbitrary ruled surfaces as in U.S. Ser. No. 16/470,885. However, the use of a rapid 2D rolling shutter sensor provides significant advantages over the line sensor design. In some aspects, the light curtains have a refresh rate of 60 fps, a 10× improvement over the prior art design, allowing the curtains to be changed rapidly and adaptively. In other aspects, the described system is more light-efficient and achieves similar range with less collected light because the optics in front of a 2D sensor are not limited by the size of a steering mirror. In yet other aspects, the described system has fewer moving parts and is more reliable. The described system of triangulation light curtains and depth sampling works outdoors at ranges up to 20-30 m and indoors at up to 50 m. A method to suppress ambient illumination (including sunlight) is also proposed as an aspect of the described embodiments.


The described system enables real-time adaptation for agile depth sensing tasks ranging from human robot interaction, to robot manipulation, path planning and navigation. The system can be reconfigurable and can greatly impact robotics and manufacturing applications.





BRIEF DESCRIPTION OF THE DRAWINGS


FIG. 1 schematically shows a top-view of the planar-parallel geometry of a triangulation light curtain using 2D cameras.



FIG. 2 shows a light curtain design with 2D cameras.



FIG. 3 shows the results of a method of ambient light suppression.



FIG. 4 shows that captured light curtains can be imaged and changed up to 60 limes per second to quickly scan different curtains through a volume.



FIG. 5 shows the described device capturing the depth of small, fast objects



FIG. 6 shows adaptive depth imaging of a table and chairs scene.



FIG. 7 shows depth imaging of a scene using planar curtains, random curtains, and adaptive curtains.



FIG. 8 shows the results of a mobile robot being driven through a highbay scene.





DETAILED DESCRIPTION

Light curtains can be imaged by steering a line imager with a galvomirror, but the nature of that design can limit the frame rate to 5.6 fps and may require the use of a small lens to fit the line sensor's field-of-view onto the rotating galvomirror. This may reduce its light efficiency and range. Both of these issues can be improved by using a 2D camera and larger aperture lens to image the light curtains.


Instead of the continuous scanning of a single line sensor plane, a 2D camera can have a discrete set of imaging planes that are defined by the optics and pixel array. In at least some designs of a light curtain, the intersection of the pixels on these planes and the light curtain surface can be found. For simplicity of design, in some embodiments, it can be assumed that the pixel rays of a given camera column are co-planar and that the light sheet projector emits a true plane of light and that its rotation axis is parallel to columns of the camera. These assumptions can be generally enforced by using a low distortion lens, careful design of the light sheet projector optics, and precise alignment of the projector axis to the camera columns. With these assumptions, light curtains can be designed in two-dimensions by looking at the rays on the XZ-plane, as shown in FIG. 1. Given that a 3D point, Xi is on the intersection of the camera plane, pci, and the desired curtain surface. The required projected light plane, ppi, and angle, θp,i, to image that point can be found through simple geometry by creating a plane from Xi and two points that lie on the camera plane's rotation axis (e.g. (0, 0, 0) and (0, 1, 0)). As the camera plane changes (as indicated by the faint lines), the required light planes are calculated for each new point. The required laser plane may be found in a similar manner by first projecting Xi; into the frame of the projector to get:

Xip=TcpXi  (1)

where Tcp is the transformation matrix that converts points in the camera frame to the projector frame found through calibration.


The point Xip is then used with two points on the projector rotation axis to find the projected light plane, ppip, which is in the frame of the projector. This plane in the camera frame is then found by Eq. (2):

ppi=Tcp⊥ppip  (2)


To image a curtain profile, the desired profile is first discretized into a series of m points uniformly distributed along the curtain profile, where m is approximately the number of columns of the camera. Line segments are then formed between adjacent points. The ray representing each column is then checked for intersection with the line segments to produce a series of points, Xi . . . Xn on the column rays that lie on the light curtain profile, as shown in FIG. 2. It is possible that some of the camera rays will not intersect the desired curtain profile or that the design points will be outside the field-of-view of the light sheet projector or camera. Any points that lie outside the field-of-view of either module may be marked invalid and not used to image the light curtain. If the design point is valid, it is transformed into the frame of the light sheet projector using Eq. (1) and the galvomirror angle needed to create a light sheet that will travel through the design point is calculated using Eq. (3):

θp,i=a tan 2(xz,ip,xx,ip  (3)


For an ideal system with true planes of illumination and imaging, the 3D points of the surface can be defined by the intersection line of the two planes. However, for a real system, the lens may have a small amount of distortion and there may be some small alignment error between the camera and light sheet projector. In this manner, the true points of intersection with the light sheet plane for pixels in a given column may not be co-planar and can be found by calculating the intersection of each pixel ray with the light sheet plane, using known calculation methods for ray-plane intersection. This means that the actual 3D surface of the light curtain can vary from the designed profile throughout the camera field of view as shown in FIG. 2. View (A) shows light curtains designed with a 2D camera by finding the points at the intersection of the camera rays with the desired curtain profile. This produces a non-uniformly spaced set of points on the curtain surface. Due to small optical distortion and alignment error, the actual 3D profile of the curtain is not a perfect extrusion of the designed curtain into 3D, as shown by the difference in curtain shape along each axis in View (B). For example, note the varying depth of the front of the curtain in the z-axis. However, depending on the severity of the distortion and misalignment, the curtain profile may be fairly consistent towards the middle of the field-of-view and only changes significantly towards the edges of the field-of-view.


Fast Curtain Imaging with Rolling Shutter Cameras


Light curtain imaging can require the imaging plane to change and intersect the light curtain surface where desired. For a 2D camera, this can be done by capturing an entire frame and only using a given column (slow) or by only imaging a select region-of-interest on the imager (faster, but still slow). Although simple to implement, both of these methods may be too slow to enable agile light curtain imaging.


A much quicker method of imaging the curtain can involve using the rolling shutter of a 2D CMOS imager to move the imaging plane. The rolling shutter can change the imaging plane rapidly at a uniform speed specified by the pixel clock of the imager. This characteristic can enable light curtain imaging at the full frame rate of the rolling shutter camera. Imaging a curtain can be as simple as commanding the light sheet projector to the angle necessary to project the light sheet at the point defined by the intersection of the active row of the 2D camera and the light curtain profile. By synchronizing the motion of the rolling shutter with the motion of the light sheet, light curtains forming any ruled surface can be imaged.


For each frame of the rolling shutter camera, an image of the captured curtain can be produced. While imaging light curtains, the camera captures both laser light and ambient light. If the ambient light is low enough, (i.e. indoor imaging), the image from the camera can be directly thresholded to produce a mask indicating the detected points. However, in many circumstances the captured ambient light is much greater than the captured laser light and the curtain cannot be detected (i.e. outdoors in sunlight). A narrow band-pass filter significantly reduces the captured ambient light, but for maximum performance it is desirable to detect the laser signal at as few bits as possible.


Ambient Subtraction


To increase the performance in the presence of ambient light, ambient light may be subtracted by capturing both an ambient image and a combined laser+ambient image at each light curtain position. In this example, the ambient image is subtracted from the combined image to get an image with just the curtain light. This allows for greater ambient performance but requires that the same camera plane be imaged twice. This can be done with a 2D camera if the entire image or a selectable region of interest is being imaged, but in some instances may not be done with a single rolling shutter frame capture. To solve this, a method of the present invention, in some embodiments, uses adjacent columns of a captured image to perform ambient subtraction.


The method may sacrifice curtain resolution and may capture a raw image where even columns are captured with the laser on and odd columns with the laser off to get an ambient-only image and a combined image, as shown in FIG. 3. Separating these columns and upsampling them forms a full resolution ambient-only image and a combined image. Subtracting the ambient image from the combined image produces a laser-only image, which can then be thresholded to find the light curtain detections, providing a mask indicating the imaged detection points. Filtering this mask with a thin horizontal erosion/dilation filter provides improved robustness to large intensity gradients and upsampling artifacts.


In areas of high ambient light, this technique may still produce errors at locations of high intensity gradient as shown by the faint edges in the laser image and vertical thin lines in the thresholded mask image in FIG. 3.


Depending on the synchronization precision of the rolling shutter motion with the motion and timing of the light sheet projector, there may be slight bleed-through of the laser light onto neighboring columns which may show up in the ambient image. Because this light may appear in the ambient image, it can be subtracted from the combined image and the measured signal from the laser light can be reduced, thereby reducing the performance of the device. With a precise synchronization, the amount of this light can be limited to a few bits or removed altogether, so that it does not greatly affect the performance of the system.


Limitations


In some embodiments, in exchange for the imaging speed and enhanced light efficiency of using a rolling shutter camera, the uniform sampling of the light curtain profile that dual galvomirrors provides prior art designs may be surrendered due to the discrete nature of the camera pixels. This can lead to a situation where there may be a gap in the curtain profile that the light curtain device cannot image.


This may occur when the rays of the camera are similar in direction to the curtain surface. This effect is shown in the top-right portion of the curtain profile in FIG. 2. One other disadvantage of using a rolling shutter camera may, in some examples, be that each plane of the camera can only be imaged once in a given frame. If a camera plane intersects multiple curtain segments (e.g. a zig-zag) one of the segments can be chosen for the imaged curtain and sequential curtains can be imaged to capture all the points along a given ray. One limitation of the galvomirror may be that, if it is commanded to move very quickly (greater than a 100 Hz step function), then it may lag the commanded position and the intersection of the planes will be at a point different than designed, which may in turn cause errors in the locations of the detected points. This may constrain which curtains can be imaged and, in practice, may require random and adaptive curtains to be smoothed with a spline before imaging. This error can be accounted for by using the angle measured from the closed-loop feedback of the mirror lo calculate the position of the detected points, rather than trusting open-loop control.


Hardware Prototype


A hardware prototype of the system comprises of a light sheet projector and a rolling shutter camera. The described system is exemplary only, and embodiments of the system covered by the claims are is not meant to be limited to the described components. The light sheet projector may contain a custom-designed line laser module using a 1 W 830 nm laser diode that is then collimated and shaped into a line with a 45° Powell lens. This laser line is then projected onto and steered with a galvomirror. The line laser module is aligned and mounted to the galvomirror in a mount that enables the precise collinear alignment of the laser line with the galvomirror's axis of rotation. The galvomirror may have dimensions of 14.5 mm×8.5 mm and may have a 50° optical scan angle.


The rolling shutter camera may be fitted with a low distortion C-mount lens with 70° (h)×60° (v) field of view. This camera may provide 10-bit precision and may have a native resolution of 1280×1024 pixels with 5.3 μm square pixels. The camera may be operated in 2× binned mode for a resolution of 640×512 to increase the signal of the received light and reduce the noise. A low-distortion lens may be used to ensure that the intersection of the light sheet and the camera rays along a given column form as close to a line as possible. A 12 nm bandpass filter may be placed centered at 830 nm between the lens and the image sensor to reduce the amount of collected ambient light. The camera may be aligned to the galvomirror such that the rows of the camera are parallel with the galvomirror's axis of rotation. The rotated camera may then be placed at a fixed baseline of 200 mm from the galvomirror's rotation axis. A micro-controller may be used to synchronize the camera, the laser, and the galvomirror. A color 2D helper camera may be used for visualizing the light curtains and detecting results in the scene by projecting the light curtain to its view.


In alternate embodiments, different types of sensors may be used, including, but not limited to CMOS (complementary metal oxide semiconductor), InGaAs (indium gallium arsenide), SWIR (short-wave infrared), DVS (dynamic vision sensor), ToF (time of flight) and CW-ToF (continuous wave time of flight) type sensors.


Calibration


The performance of light curtains depends on precise calibration of the camera and light sheet projector. First, the camera intrinsics may be determined. Next, the extrinsic calibration of the light sheet projector and camera may be determined by imaging a set of light planes projected by the light sheet projector onto a planar wall. A checkerboard target of known dimensions may be attached to the wall and imaged with the calibrated camera to get the known 3D coordinates of points on each imaged laser line. This may be repeated with the same set of planes at several depths to fully define each plane of light. Best fit plane equations may then be found for each set of points using weighted least squares where the weights were the normalized intensity values. Then, given the equations for the planes, the location of the galvomirror axis with respect to the camera may be found by a least squares fit of a line to the intersection of all the planes. A continuous function may be fitted to the relationship of the plane angle (with respect to the galvomirror axis) and the commanded light sheet projector angle, which is then used to determine the commanded galvomirror position needed for any speci-fied angle of a given design point.


Capture Process


In prior-art light curtain designs, the light sheet and camera plane angles were commanded by the positions of galvomirrors. With a 2D camera, however, the camera plane angle is defined by the optics and the active line of pixels on the imager. For a rolling shutter camera, the active line of pixels is defined by the speed of the rolling shutter and the time since the start of the frame. The speed of the rolling shutter is determined by the readout rate of the pixels, known as the pixel clock. The maximum time that a given line is active can be found by dividing the number of pixels on the line by the pixel clock. At the maximum 60 fps capture rate of the camera, the maximum active exposure lime of a line is ≈15 μs.


To image a light curtain, the software causes the processor to calculate the required galvomirror positions and transmit them to a micro-controller controlling the galvomirror. At the same time, the camera is triggered to start the frame capture. The software then sequentially commands the galvomirror positions and laser power in lock step with the timed progression of the rolling shutter. This process then repeats for each successive light curtain. Calculating the galvomirror positions for every frame allows for a different light curtain to be imaged every frame at the full frame rate of the camera.


Working Range


At some frame rates, the described device has a working range of 20 m while imaging a whiteboard in approximately 50 klx of ambient light and a working range of 50 m indoors. This has a reduced exposure time. When the device is configured for an exposure of 100 μs, the device can image over 40 meters outdoors in similar conditions.


Agile and Dynamic Light Curtains


The described device is capable of imaging 60 different light curtains per second. This speed and flexibility enables the use of agile and dynamic light curtains that can be used to intelligently and adaptively sample the scene. This capability can be applied to many areas including path planning, high-resolution safety curtains, and depth sensing. FIG. 4 shows the results of imaging various types of light curtains both indoors and outdoors and are just a sample of the different types of curtains that can be imaged with the described device. Images shown in FIG. 4 are the light curtain surface and detections projected into the helper camera's view for visualization on the scene. The images shown in FIG. 4 are from the 2D helper camera's view with the light curtain surface rendered in blue and detections rendered in green. The agile nature of these curtains enables many applications including the rapid checking of planned paths, shown in View (A), and detecting obstacles entering and within a safety zone by rapidly alternating between checking the border of the safety-zone and the area within it, shown in View (B).


High-Resolution, High Speed Light Curtains:


The rapid capture rate and resolution of the described device enables the imaging of small and fast objects as they pass through light curtains, as shown in Views (A-C) of FIG. 5. Several composite images show the view and detected location of small 70 mm diameter balls in View (A) and 30 mm diameter balls and a 30 mm thick, 265 mm diameter Frisbee that was thrown through a planar curtain 5 m away in in View (B). The detections of the objects are shown in View (C). The resolution of light curtains provides increased detail over scanning LIDAR devices and can enable enhanced object recognition and critical detection of small objects (e.g. wires, branches, etc). This is especially noticeable when imaging thin structures or objects at a distance as shown in Views (D-I) of FIG. 5. Views (D-F) shows a planar curtain swept through a volume containing a thin wire fence to create a dense 3d pointcloud in View (E). The light curtain (green points) in View (F) reconstructs the fence mesh at much higher resolution than a 16-beam Velodyne VLP-16 scanning LIDAR (white points) at a distance of 1.5 m away. By imaging a planar curtain, the range and resolution of the light curtain enable it to create high resolution height maps of objects at ranges of 15 m outdoors, as shown in Views (G-I). At this range, a static VLP-16 scanning LIDAR only senses the objects as a few points, shown in white.


Adaptive Depth Imaging


The ability to specify depths of interest at high-rates enables intelligent depth imaging of an environment based on the current knowledge of the scene. For example, when a device first enters into the scene, it has no knowledge of the environment, but by quickly and randomly scanning a volume of interest with light curtains it can generate a coarse estimate of the locations of objects in the scene, and one or more new curtains can be designed to image around these points-of-interest. This process is then rapidly repeated to form an accurate map of the objects in the scene. FIG. 6 shows adaptive depth imaging of a scene containing a table and chair. The imaging is performed by first randomly sampling a volume of interest to find objects. Then, the detected points at the front of these objects (indicated by red dots in plots and white dots in bottom row) are used to design a set of light curtains which is then imaged to refine and discover more of the scene. As time continues, the curtains eventually discover and image the back wall of the scene. FIG. 6, View (A) shows random curtains used to initialize the curtain by randomly sampling a volume of interest within 3 m of the light curtain. These curtains detected several objects, which the device then used as design points to fit a new curtain to the front surfaces of the objects. In less than a few milliseconds, a set of 10 curtains were designed that were scaled versions of this spline to cover the area directly in front of and behind the detected scene points. These new curtains were then imaged and used to refine and discover more of the scene. By interleaving a few random curtains with the adaptive curtains, the device can continue checking the scene for any changes and sample the rest of the scene at low resolution. The design process for the experiment in FIG. 6 projected all of the detected points to the XZ-plane and used the closest point within an angular region as the design point for that region. By splitting the entire plane into these uniformly spaced angular regions, a set of design points were determined.


Depth Imaging Comparison


A comparison of depth imaging methods using different light curtain types was performed by capturing the same scene with each curtain type and comparing the coverage of the scene at specified times. The methods included plane sweeping, random sampling, and adaptive depth imaging. For plane sweeping, the planes were designed to fully image the 4.25 m scene in 1.0 with a depth resolution of 0.2 m. The adaptive curtains were tuned to sense the front surfaces of detected objects at a high resolution and the rest of the scene at a low resolution. FIG. 7 shows the results of this comparison. Once initialized with 0.25 s of random detections, the discovery nature of the adaptive curtain enabled it to sense near the detected objects and not waste time sensing empty areas at the front of the scene. As a result, it was able to quickly cover the interesting parts of the scene in less than 0.25 s. Given the time, plane sweep curtains can provide high-resolution and complete coverage, but with limited time, random and adaptive curtains can image more of the scene in less time. For example, the plane sweep curtains could have been configured to image the entire scene in 0.5 s but at 0.4 m depth resolution, which is much less than the other methods. FIG. 7 shows that adaptive curtains intelligently cover more of the scene in less time than sweeping a plane or random sampling of the scene. After initialization with random sampling, the adaptive curtains discovered the interesting parts of the scene in 0.25 s and then continued to refine the map.


Discovery and Mapping Using Adaptive Light Curtains


Adaptive light curtains can be used to discover and map a scene from a moving platform. A small robot with on-board localization was used to move the light curtain device through a cluttered highbay scene, as shown in FIG. 8. As the robot progressed through the scene, adaptive light curtains discovered the scene structure and continuously adapted to image the newly detected objects from each frame. Rather than using a fixed depth sensing strategy, the light curtains intelligently sampled regions around the detected objects at high-resolution and sampled the rest of the scene with random curtains at a lower resolution. For our mapping experiments, a set of 5 uniquely random curtains were interleaved with every 10 adaptive curtains. FIG. 8 shows a few instances of the curtains fitting to objects in the scene. Notice that the curtains, shown in blue in View (A), fit tightly around the objects in the scene. When moved throughout the environment, the detected points on the curtains, shown in green in View (B) also mapped the scene. The white points on the 3D maps show the design points for the set of adaptive curtains.


To those skilled in the art to which the invention relates, many modifications and adaptations of the invention will suggest themselves. The exemplary methods and systems disclosed herein are not to be taken as limitations on the invention, but as an illustration thereof. The intended scope of the invention is defined by the claims which follow.

Claims
  • 1. A system comprising: a light source for creating a light sheet plane;a line sensor having an adjustable region of interest for sensing a line in a sensing plane, the sensed line being defined by an intersection of the light sheet plane and the sensing plane;a processor; and software executed by the processor for performing the functions of:sampling a scene along a non-uniform plurality of sensed lines in the scene;discovering one or more objects in the scene at one or more of the plurality of sensed lines;increasing the density of the sensed lines in areas of the scene containing the one or more discovered objects to create one or more adaptive curtains of sensed lines.
  • 2. The system of claim 1 wherein the light source comprises: a laser diode;a collimation lens;a Powell-type lens; anda steerable galvomirror;wherein light emitted by the laser diode is formed into a line of light by the Powell-type lens and before striking the steerable galvomirror.
  • 3. The system of claim 1 wherein the line sensor comprises a camera having a rolling 2D shutter.
  • 4. The system of claim 1, wherein discovering of the one or more objects comprises detecting a variance in light detected by the line sensor, indicative of the presence of an object intersecting the sensed line.
  • 5. The system of claim 4 wherein the discovering of the one or more objects further comprises: determining a position and depth for each of the one or more discovered objects using triangulation based on positions of the light sheet plane and the sensing plane.
  • 6. The system of claim 1, the non-uniform plurality of sensed lines being sparsely distributed within the scene.
  • 7. The system of claim 1, the non-uniform plurality of sensed lines being randomly distributed within the scene.
  • 8. The system of claim 2 wherein the camera has a pixel array comprising a plurality of columns of pixels and further wherein the rotation axis of the galvomirror is parallel to the plurality of columns of pixels.
  • 9. The system of claim 8, the increased density of the sensed lines in areas of the scene containing the one or more discovered objects defining a light curtain profile, the light curtain profile being imaged by: discretizing the light curtain profile by defining a plurality uniformly distributed of points along the profile;calculating, for each point, an angle for the galvomirror to cause the light sheet plane to intersect the point; andcapturing light reflected from an object intersecting a sensed line at each defined point.
  • 10. The system of claim 9 wherein each defined point corresponds to a single column in the pixel array of the camera.
  • 11. The system of claim 10, wherein capturing light reflected from a sensed line at each point further comprises: calculating, for each pixel in the single column of pixels that correspond to the defined point currently illuminated by the light sheet plane, the intersection of a pixel ray with the light sheet plane.
  • 12. The system of claim 9 further comprising, for each sensed line: capturing a first image comprising only ambient light;capturing a second image comprising both ambient light and illumination from the light source; andsubtracting the first image from the second image.
  • 13. The system of claim 9 wherein the software further causes the processor to: calculate a required galvomirror position and transmit it to a micro-controller controlling the galvomirror;trigger the camera to start a frame capture; andsequentially command galvomirror positions in lock step with a timed progression of the rolling shutter of the camera.
  • 14. The system of claim 13 wherein the software further causes the processor to: control the laser light diode with a desired laser power setting.
  • 15. The system of claim 1 wherein one or more adaptive light curtains can be created to track one or more objects in the scene while simultaneously sampling the scene with non-uniform, sparse sensed lines.
  • 16. A computer-implemented method comprising: sampling a scene along a non-uniform plurality of sensed lines in the scene, each sensed line defined as an intersection of a light sheet plane and a sensing plane;discovering one or more objects in the scene at one or more of the plurality of sensed lines;increasing the density of the sensed lines in areas of the scene containing the one or more discovered objects to create one or more adaptive curtains of sensed lines, each adaptive curtain adapted to sense one or more of the discovered objects.
  • 17. The method of claim 16 wherein the discovering of the one or more objects comprises: determining a position and depth for each of the one or more discovered objects using triangulation based on positions of the light sheet plane and the sensing plane.
  • 18. The method of claim 16, the non-uniform plurality of sensed lines being sparsely and randomly distributed within the scene.
  • 19. The method of claim 16 further comprising, for each sensed line: capturing a first image comprising only ambient light;capturing a second image comprising both ambient light and illumination from the light source; andsubtracting the first image from the second image.
  • 20. The method of claim 16 wherein one or more adaptive light curtains can be created to track one or more objects in the scene while simultaneously sampling the scene with non-uniform, sparse sensed lines.
RELATED APPLICATIONS

This application is a national phase filing under 35 U.S.C. § 371 claiming the benefit of and priority to International Patent Application No. PCT/US2019/052854, filed on Sep. 25, 2019, which is a continuation-in-part of U.S. patent application Ser. No. 16/470,885, which is a national phase filing under 35 U.S.C. § 371 claiming the benefit of and priority to International Patent Application No. PCT/US2019/021569, filed on Mar. 11, 2019, which claims the benefit of U.S. Provisional Patent Application Ser. No. 62/761,479, filed Mar. 23, 2018. Further, U.S. patent application Ser. No. 16/470,885 is a continuation-in-part of U.S. patent application Ser. No. 15/545,391, which is a national phase filing under 35 U.S.C. § 371 claiming the benefit of and priority to International Patent Application No. PCT/US2016/017942, filed on Feb. 15, 2016, which claims the benefit of U.S. Provisional Patent Application Ser. No. 62/176,352, filed Feb. 13, 2015. Further, this application also claims the benefit of U.S. Provisional Patent Application Ser. No. 62/920,178, filed Apr. 17, 2019. The entire contents of these applications are incorporated herein by reference in their entirety.

GOVERNMENT INTEREST

This invention was made with government support under contracts HR0011-16-C-0025, awarded by the Defense Advanced Research Projects Agency (DARPA) and CNS-1446601, awarded by the National Science Foundation (NSF). The government has certain rights in the invention.

PCT Information
Filing Document Filing Date Country Kind
PCT/US2019/052854 9/25/2019 WO
Publishing Document Publishing Date Country Kind
WO2020/214201 10/22/2020 WO A
US Referenced Citations (96)
Number Name Date Kind
4621185 Brown Nov 1986 A
4687325 Corby, Jr. Aug 1987 A
4687326 Corby, Jr. Aug 1987 A
5128753 Lemelson Jul 1992 A
5717390 Hasselbring Feb 1998 A
5852672 Lu Dec 1998 A
6043905 Kato Mar 2000 A
6529627 Callari et al. Mar 2003 B1
6556307 Norita et al. Apr 2003 B1
7978928 Tan et al. Jul 2011 B2
8339616 Teodorescu Dec 2012 B2
8400494 Zalevsky et al. Mar 2013 B2
9500477 Bunel et al. Nov 2016 B2
9536320 Prince Jan 2017 B1
9838611 Haraguchi Dec 2017 B2
10021284 Wang et al. Jul 2018 B2
10145678 Wang et al. Dec 2018 B2
20010035636 Adachi Nov 2001 A1
20020014533 Zhu et al. Feb 2002 A1
20040151345 Morcom Aug 2004 A1
20050128196 Popescu Jun 2005 A1
20050219552 Ackerman Oct 2005 A1
20060132752 Kane Jun 2006 A1
20070242872 Rudin et al. Oct 2007 A1
20070268398 Raskar Nov 2007 A1
20080118143 Gordon May 2008 A1
20080123939 Wieneke May 2008 A1
20090066929 Tropf Mar 2009 A1
20090201486 Cramblitt et al. Aug 2009 A1
20100046801 Ishiyama Feb 2010 A1
20100074532 Gordon et al. Mar 2010 A1
20100085425 Tan Apr 2010 A1
20100128221 Muller et al. May 2010 A1
20100303299 Cho et al. Dec 2010 A1
20110071675 Wells Mar 2011 A1
20110102763 Brown et al. May 2011 A1
20110235018 Mori et al. Sep 2011 A1
20110292347 Zhang et al. Dec 2011 A1
20110299135 Takabatake Dec 2011 A1
20110317005 Atkinson Dec 2011 A1
20120008128 Bamji Jan 2012 A1
20120062705 Ovsiannikov et al. Mar 2012 A1
20120062963 Gillham et al. Mar 2012 A1
20120200829 Bronstein et al. Aug 2012 A1
20130010087 Nieten et al. Jan 2013 A1
20130021474 Taylor et al. Jan 2013 A1
20130121560 Hirai May 2013 A1
20130127854 Shpunt et al. May 2013 A1
20130250272 Ludwig Sep 2013 A1
20140002610 Xi Jan 2014 A1
20140022251 Aila Jan 2014 A1
20140055771 Oggier Feb 2014 A1
20140055779 Enami Feb 2014 A1
20140111616 Blayvas Apr 2014 A1
20140125775 Holz May 2014 A1
20140211193 Bloom Jul 2014 A1
20140232566 Mimeault et al. Aug 2014 A1
20140247323 Griffis et al. Sep 2014 A1
20140253724 Yamagata et al. Sep 2014 A1
20140327746 Dubois Nov 2014 A1
20140328535 Sorkine-Hornung Nov 2014 A1
20150067929 Blanton et al. Mar 2015 A1
20150176977 Abele et al. Jun 2015 A1
20150177506 Nishiwaki Jun 2015 A1
20150215547 Muller Jul 2015 A1
20150229915 Kirk Aug 2015 A1
20150253428 Holz Sep 2015 A1
20150281671 Bloom et al. Oct 2015 A1
20150285618 Haraguchi Oct 2015 A1
20150285625 Deane Oct 2015 A1
20150294496 Medasani et al. Oct 2015 A1
20150362698 Lansel Dec 2015 A1
20150381962 Song Dec 2015 A1
20160041266 Smits Feb 2016 A1
20160065945 Yin et al. Mar 2016 A1
20160124203 Ryu May 2016 A1
20160182788 Wan Jun 2016 A1
20160182889 Olmstead Jun 2016 A1
20160198147 Waligorski et al. Jul 2016 A1
20160209183 Bakken et al. Jul 2016 A1
20160307325 Wang Oct 2016 A1
20160335778 Smits et al. Nov 2016 A1
20160349369 Lee et al. Dec 2016 A1
20170064235 Wang et al. Mar 2017 A1
20170127036 You et al. May 2017 A1
20170142406 Ovsiannikov et al. May 2017 A1
20170272726 Ovsiannikov Sep 2017 A1
20170310948 Pei et al. Oct 2017 A1
20170353707 Wang et al. Dec 2017 A1
20170366801 Horesh Dec 2017 A1
20180246189 Smits Aug 2018 A1
20180252800 Morcom Sep 2018 A1
20180307310 Mccombe et al. Oct 2018 A1
20180374230 Narasimhan Dec 2018 A1
20190025986 Yamauchi Jan 2019 A1
20190236796 Blasco Claret et al. Aug 2019 A1
Foreign Referenced Citations (14)
Number Date Country
H08159756 Jun 1996 JP
H11132713 May 1999 JP
2002039716 Feb 2002 JP
2004150810 May 2004 JP
2010203864 Sep 2010 JP
2012168049 Sep 2012 JP
2013050310 Mar 2013 JP
2015513825 May 2015 JP
2016091006 May 2016 JP
03016982 Feb 2003 WO
2015003108 Jan 2015 WO
2015119872 Aug 2015 WO
2016131036 Aug 2016 WO
2018136709 Jul 2018 WO
Non-Patent Literature Citations (24)
Entry
Wang et al., Programmable Triangulation Light Curtains, 2018, ECCV 2018 LNCS, SpringerLink: https://link.springer.com/conference/eccv (Year: 2018).
J. Bartels, J. Wang, W. Whittaker and S. Narasimhan, “Agile Depth Sensing Using Triangulation Light Curtains,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 7899-7907, doi: 10.1109/ICCV.2019.00799 (Year: 2019).
S. J. Koppal and S. G. Narasimhan, “Novel Depth Cues from Uncalibrated Near-field Lighting,” 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, 2007, pp. 1-8, doi: 10.1109/ICCV.2007.4408900. (Year: 2007).
European Search Report and Written Opinion for Application No. EP19924858, dated Nov. 24, 2022, 9 pages.
Extended European search report for EP Application No. 18742111.0, dated Jul. 10, 2020, 6 pages.
Intemational Search Report and Written Opinion for International application No. PCT/US2018/014369, dated Apr. 26, 2018, 8 pages.
Communication pursuant to Article 94(3) EPC for European Patent Application No. 18742111.0, dated Dec. 16, 2021, 5 pages.
Extended European search report for Application No. EP19772623.5, dated Oct. 22, 2021, 7 pages.
International Search Report and Written Opinion for International Application No. PCT/US2019/21569, dated May 24, 2019.
Achar, S., et al., “Epipoiar Time-of-Flight Imaging”, ACM Transactions on Graphics, 36(4):37:1-37:2, Jul. 4, 2017.
Wang, J., et al., “Programmable Triangulation Light Curtains”, ECCV Computer Vision Foundation, [online] <URL: http://openaccess.thecvl.com/content_ECCV _ 2018/papers/Jian_ Wang_ Programmable_Lght_ Curtains_ECCV _ 2018_paper.pdf>, pp. 1-14 (2018).
O 'Toole, M., et al., “Homogeneous Codes for Energy-Efficient Illumination and Imaging” (2015).
Tadano, R., et al., “Depth Selective Camera: A Direct, On-chip, Programmable Technique for Depth Selectivity in Photography”, IEEE International Conference on Computer Vision (2015).
Blais, F., “Review of 20 Years of Range Sensor Development”, Journal of Electronic Imaging, 13(1): 231-240, Jan. 2004.
Heckman, P. J., “Underwater Range Gated Photography”, Proc. SPIE 0007, Underwater Photo Optics I, Jun. 1, 1966, [online] https://www.spiedigitallibrary.org/conference-proceedings-of-spie on Jan. 28, 2019.
Jarvis, R. A., “A Perspective on Range Finding Techniques for Computer Vision”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-5, No. 2, pp. 122-139, Mar. 1983.
Barry, A.J., et al, “Pushbroom Stereo for High-Speed Navigation in Cluttered Environments”, IEEE International Conference on Robotics and Automation (ICRA), pp. 3046-3052, (2015).
International Search Report and Written Opinion for International Patent Application No. PCT/US16/17942, dated May 19, 2016.
International Preliminary Report on Patentability for International Patent Application No. PCT/US16/17942, dated Aug. 15, 2017.
O'Toole et al. “3D Shape and Indirect Appearance by Structured Light Transport.” University of Toronto. 2014 (2014), pp. 1-3, 6-7 [on line] <URL: http://www.cs.toronto.edu/-kyros/pubs/14.cvpr .slt.pdf>.
O'Toole et al. “Prima-Dual Coding to Probe Light Transport.” ACM. 2012 (2012), pp. 39:1-39:6 [online] <URL:http://www.cs.toronto.edu/-kyros/pubs/12.sig.pdc.pdf>.
International Search Report and Written Opinion for International application No. PCT/US19/052854 dated Jul. 15, 2020, 9 pages.
S. B. Gokturk, H. Yalcin and C. Bamji, “A Time-of-Flight Depth Sensor—System Description, Issues and Solutions,” 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA, 2004, pp. 35-35, doi: 10.1109/CVPR.2004.291. (Year: 2004).
Horaud et al “An overview of depth cameras and range scanners based on time-of-flight technologies” Machine Vision and Applications—arXiv:2012.06772 [cs.CV] (2016).
Related Publications (1)
Number Date Country
20220207761 A1 Jun 2022 US
Provisional Applications (3)
Number Date Country
62920178 Apr 2019 US
62761479 Mar 2018 US
62176352 Feb 2015 US
Continuation in Parts (2)
Number Date Country
Parent 16470885 US
Child 17601780 US
Parent 15545391 US
Child 16470885 US