Patent Application
20210109197
All publications and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.
In digital photography a charge-coupled-device CCD sensor can gather light from several million directions simultaneously to generate detailed images. In contrast, many light detection and ranging systems (LIDARs) scan or rotate laser beams to measure the time of flight in a sequence of directions. The sequential measurement nature limits the total number of range measurements per second. Hence a LIDAR that scans a FOV in a uniform deterministic manner can provide poor angular resolution. In a related area analog micromirror arrays have been proposed for providing zoom properties in digital cameras. Zooming in (e.g., narrowing the FOV) to enhance image quality can be effective for both 2D photography and 3D time-of-flight cameras (e.g., Flash LIDARs). However there are circumstances where a wide field of view and enhanced image quality are both desirable. U.S. Pat. No. 9,383,753 to Templeton discloses a LIDAR with dynamically adjustable angular resolution, but only describes dynamic angular velocity in a single axis for a rotating LIDAR. U.S. Pat. No. 9,383,753 further assumes a rotating LIDAR and does not provide for arbitrary laser orientation within a scan. Hence, dynamically adapting LIDAR or camera measurement density within a scan, to improve the accuracy of object boundary detection in the FOV remains a challenge.
Laser light poses several safety risks to humans, based on the coherent nature of laser radiation. The potential for eye damage is often the modality that requires the most stringent limits on laser power. In controlled environments (e.g. a laboratory) precautions can be used such as protective eyewear or housing a laser in a specialized enclosure with safety interlocks. In open environments (e.g. streets and highways) such precautions cannot be assumed and hence eye-safety is often ensured by using inherently eye-safe lasers (e.g. ANSI Z136.4 class 1 lasers).
Laser range finding is a useful technology for autonomous vehicles but must operate safely in human-filled environments. Maximum measurement range can benefit from higher laser intensity. However, many countries and regions of the world impose varying limits on the maximum permissible laser radiation (e.g. energy per square centimeter or energy per pulse). Traditionally, adherence to these laser radiation limits is ensured by design and validated during the laser system qualification. This designed-in approach to limiting laser radiation exposure is conservative and often suboptimal. Recent, alternative approaches attempt to sense objects in the vicinity of a laser that is operating above an intrinsically safe (e.g. eye-safe) threshold. The intensity of a laser beam can decrease as it travels from a source and hence it may only be necessary to monitor for objects (e.g. people) within a threshold distance from the source to ensure safe laser operation. U.S. Pat. No. 9,121,703 issued to Droz discloses using a proximity sensor to sense an object within a threshold distance of the laser range finder and discontinuing laser emission upon detection. Proximity sensors (e.g. passive infrared sensors) are useful for identifying objects in the vicinity but provide little specificity regarding location and the path or trajectory of objects in the field of view (FOV) of the laser system. Proximity-based laser-deactivation can be useful when a laser system emits high-intensity laser light in a wide range of azimuthal directions (e.g. 360 degrees) but can be overly-conservative (e.g. produce many false positives) for a laser system that emits high-intensity pulses in only a narrow range of directions.
U.S. Pat. No. 8,948,591 to Scherbarth discloses a laser range finder that detects objects within a threshold distance during some previous time period and discontinues laser emission upon detecting an object within the threshold distance. This approach does not address the challenge of high-intensity laser pulses during the discovery of a new object within the threshold distance. Several safety standards (e.g. ANSI Z136.4) require all laser pulses meet an eye-safe intensity requirement, even a single laser pulse during discovery of a new object.
Therefore, an ongoing technical challenge is the operation of a laser range finder in a high-intensity mode while ensuring safety and avoiding frequent false positive laser power reductions.
In one aspect a micromirror array can act like an electronically controllable transfer function for light, between an input lens of a camera or LIDAR and a photodetector array. For example an analog micromirror array can perform a zoom function by reconfiguring some or all of the micromirrors to deflect light rays from a portion of an available FOV onto the photodetector array while simultaneously spreading the portion over more elements of the photodetector. This has the effect of increasing image resolution (e.g., the number of photodetector elements per unit solid angle of the field of view or pixels per square degree or elements per steradian in the FOV). However reconfiguring the micromirror array to increase the resolution of a portion of a FOV can have the drawback of reducing the total angular range (FOV) measured by the photodetector array (i.e., zooming in on the scene can have the effect of increasing the resolution while decreasing the total FOV or 2D angular range sensed). While micromirror arrays can be configured into microlens, thereby enhancing image resolution, there are many times when a wide FOV (i.e., maintaining an original 2D angular range of the scene detected by photodetector array) is also desirable.
A system and method are provided to sense a specified FOV with enhanced resolution. In one embodiment a method performed by an imaging system comprises providing at an aperture a 2D field of view (FOV) from a scene to a micromirror array having a first configuration, and thereby deflecting light with the micromirror array from the FOV onto a photodetector array. The method further comprises detecting with the photodetector array a first set of light measurements spanning the FOV, processing the first set of light measurements and thereby identifying a region of interest (e.g., a region surrounding an object edge or a face), in the FOV. The set of light measurements can have a first resolution in the region of interest, based on the angular range that each element in the photodetector array receives, for example 1 light measurement or 1 photodetector element per one square degree of solid angle in the FOV. The first resolution can be based on the first configuration of the micromirror array. The method further comprises configuring the micromirror array based at least in part on the identified region of interest and thereby detecting with the photodetector array a second set of light measurements spanning the FOV with a second resolution in the region of interest that is greater than the first resolution.
In one aspect the method can conserve the size (e.g., angular range) of the original FOV, thereby keeping people and pets in the frame of the resulting 2D images and not distracting a user with an unwanted zoom effect. In another aspect the method can enhance image resolution while simultaneously conserving the original FOV; by configuring the micromirror array to compress light rays from one or more uninteresting portions of the FOV onto fewer pixels in the photodetector array (e.g., based on the first set of light measurements) and thereby enabling light rays from the region(s) of interest to be spread over more pixels to enhance the resolution. Therefore, by creating areas of sparse and denser light rays on the photodetector array simultaneously the original FOV is conserved.
In a system embodiment a processing subassembly with access to both sensor data from the photodetector array and a micromirror configuration can correct for the distortive effect of the dense and sparse zones on the photodetector array and generate an eye-pleasing output image. In another embodiment, data from sensors or sources other than the photodetector array can be used to identify the region(s) of interest. In a second embodiment a method performed by an imaging system comprises: Processing sensor data indicative from a scene in the vicinity of a micromirror array and thereby identifying a region of interest in the sensor data, wherein the micromirror array has a field of view encompassing at least some of the scene, wherein the micromirror array comprises a plurality of micromirrors with an initial configuration that deflects light from the region of interest towards a detector array and thereby provides a first resolution at the detector array for the light from the region of interest. The method further comprises reconfiguring at least a subset of the plurality of micromirrors in the micromirror array, based at least in part on the identified region of interest and thereby providing at the detector array a second resolution for light form the region of interest that is greater than the first resolution. In a third embodiment the micromirror array can be part of a ranging subassembly in a LIDAR. For example, a flash LIDAR can illuminate a field of view (FOV) with flashes of light (e.g., laser light) and gather reflections from the FOV at a photodetector array. A micromirror array can be configured based on an identified region of interest to non-uniformly spread the light reflections from the flashes of light based on the identified region of interest.
In a second group of embodiments a LIDAR performs a progressive boundary localization (PBL) method to determine the location of time-of-flight (TOF) boundaries to within some minimum angular spacing in a FOV (i.e., progressively resolve the boundaries of objects in environment local to the LIDAR). The method can generate a sequence of laser pulses, measure a corresponding sequence of laser reflections and measure a time of flight and direction corresponding to each of the laser pulse. In response to identifying a nearest neighbor pair of laser pulses within a range of directions for which the TOF difference is greater than a TOF threshold, dynamically steering the LIDAR to generate one or more intervening laser pulses with directions based on at least one of the nearest neighbor pair directions. The method can continue until all nearest neighbor pairs for which the TOF difference is greater than a TOF threshold have an angular separation (i.e., difference in directions for the laser pulses in each pair) less than a direction threshold (e.g., less than 0.5 degrees direction difference). In this way a PBL method can localize the boundary by refining the angular ranges in which large changes in TOF occur until such ranges are sufficiently small.
In third group of embodiments a method to perform extrapolation-based progressive boundary localization method (EPBL) with a LIDAR is disclosed. The method can use a LIDAR to find a first portion of a boundary in the FOV, extrapolate the direction of the boundary and thereby dynamically steer the LIDAR to scan in a second region of the FOV for the boundary. Hence the continuous and “straight-line” nature of object boundaries can be used to dynamically steer a LIDAR to scan the boundary. Similarly a classified object (e.g., a Van) can have a predicted boundary such that finding one part of the object and extrapolating or predicting a second portion of the object boundary (e.g., based on classification or a straight line edge in an identified direction) is used to dynamically steer a LIDAR scan. In one example, a LIDAR scans a first search region within a FOV, identifies a first set of locations or sub-regions of the first search regions that located on or intersected by a TOF boundary (e.g., an object edge). The exemplary EPBL method then extrapolates an estimated boundary location, outside the first search region, based on the first set of locations or sub-regions. The LIDAR then uses the estimated boundary location to configure or dynamically steer a laser within a second search region. The LIDAR can then process reflections form the second search region to determine if the boundary exists in the estimated boundary location.
Within examples, devices, systems and methods for controlling laser power or intensity in various regions of the FOV of a laser range finder are provided. In one example, a method generates high-intensity laser pulses (e.g. above an eye-safe intensity threshold) in a well-defined adaptive-intensity region of a FOV of a laser range finder. The method surrounds the adaptive-intensity region with a protective guard-region of the FOV (e.g. a guard-ring) of lower intensity (e.g. eye-safe intensity) laser pulses. A detector can detect laser reflections from the lower intensity laser pulses in the guard region and in response to sensing an object in the guard region, or entering the guard region within a threshold distance the laser range finder can subsequently reduce the intensity of laser pulses (e.g. to an eye safe intensity) within the adaptive-intensity region. The guard region can act as a safety feature, using low-intensity laser pulses to provide early and spatially accurate warning of objects likely to intersect the path of the high-intensity laser pulses thereby enabling intensity reduction.
In another example, a non-transitory computer readable storage medium having stored therein instructions that when executed by a computer device, cause the computing device to perform functions. The functions comprise dynamically steering with a steerable laser assembly at least one laser beam and thereby generating a first set of laser pulses in an adaptive-intensity region of a FOV, each with an intensity above a threshold intensity, and a second set of laser pulses in a guard region of the FOV, each with an intensity below the threshold intensity. The functions further comprise directing, based on the dynamic steering of the laser beam, the second set of laser pulses such that the guard-region adjoins or encloses at least some of the perimeter of the adaptive-intensity region. The functions can position the guard region such that a plurality of straight line paths in the plane of the FOV that enter the FOV from an edge and intersect the adaptive-intensity region, must first traverse the guard-region, thereby providing forewarning of objects (e.g. pedestrians) likely to enter the adaptive-intensity region. The functions also comprise detecting with detector a set of laser reflections corresponding to the second set of laser pulses. The function also comprise, in response to sensing a first object in the guard region, based at least in part on the set of laser reflections, generating a third set of laser pulses in the adaptive-intensity region each with an intensity below the threshold intensity.
The guard region can serve to detect objects approaching the adaptive-intensity region of the FOV and trigger the laser range finder to reduce the intensity upon detection of an object in the guard region. In one aspect, the laser pulses in the adaptive-intensity region of the FOV can be attenuated (e.g. generated at an eye-safe intensity) in response to detecting and object in the guard-region. In another aspect, a safety test can be evaluated on objects in the guard region (e.g. a criterion that determines whether an object is on a trajectory that will soon intersect the adaptive-intensity region) and the intensity of laser pulses in the adaptive-intensity region can be based on the result of the safety test. Therefore, in one embodiment the present disclosure provides a benefit over systems that discontinue or attenuate laser power in a region when an object is sensed in that region, by instead using a trajectory measured in a defined guard region to control intensity in an adaptive-intensity region. The guard region can be adjoining the adaptive-intensity region and the measured trajectory of an object can indicate imminent intrusion into the adaptive-intensity region.
In another aspect, some of the laser reflections in the guard region can come from known sources (e.g. trees or a portion of a vehicle that is always in the FOV). In one embodiment a method can define one or more mask regions of the FOV whereby reflections from objects in the mask regions are discounted in the process of evaluating a safety test on reflections from the guard region of the FOV in the process of determining the intensity of future laser pulses in the adaptive-intensity region of the FOV.
In a related group of embodiments a laser range finder can receive location estimates for a set of objects in a FOV. The laser range finder can obtain an age associated with each location estimate (e.g. the time elapsed since laser reflections associated with an object location estimate). The laser range finder can determine an object region (e.g. a portion of the FOV or a volume of space) associated with the object at a later time, based at least in part on the age of the location estimate and the position of the location estimate. The laser range finder can generate one or more laser pulses with intensities based on the object regions for the objects. For example, an object in the guard region of the FOV (e.g. a pedestrian) and moving towards the adaptive-intensity region at a slow rate of speed can cause the laser range finder to reduce intensity in the adaptive-intensity region. Conversely, a slow moving pedestrian some distance away (e.g. 100 m) may generate a much smaller object region in the FOV (e.g. angular region at some later time) and thereby not pose an imminent threat of entering or intersecting the path of high intensity laser pulses in an adaptive-intensity region of the FOV. In this case, the laser range finder can generate high-intensity laser pulses, based on the location estimate and the estimate age (e.g. the estimate is 0.5 seconds old).
In one embodiment an imaging system (e.g., a LIDAR or camera) contains a micromirror array that is configured in response to sensor data to dynamically enhance a complex shape region of interest in a field of view (FOV). The micromirror array functions as like an electronically controllable transfer function for light, between an input FOV and a detector array, thereby providing dynamically defined resolution across the detector array. Data from various configurations of the micromirror array is then combined in a 2D or 3D output image. In one aspect the imaging system begins with a first uniform resolution at the detector array and subsequently reconfigures the micromirror array to enhance resolution at a first portion of the detector array (e.g., spread an interesting object across more pixels) reduce resolution from in a less interesting part of a scene and thereby sample all of the original FOV with anisotropic resolution.
In one embodiment a LIDAR generates high-intensity laser pulses with intensities above a threshold intensity (e.g. above an eye-safe intensity) in a 2-D angular range in a field of view. The LIDAR further generates low-intensity (e.g. eye-safe) laser pulses in a protective guard region (e.g. a guard ring) that surrounds the high-intensity laser pulses. In response to detecting an aspect of an object using reflections from the low-intensity laser pulses (e.g. a person on a trajectory that will intersect the high-intensity laser pulses) the LIDAR modifies the angular range of subsequent high intensity laser pulses. In this way the LIDAR can adapt or steer the angular range of the high-intensity laser pulses to avoid an object detected within the low-intensity guard region.
The techniques described in this specification can be implemented to achieve the following exemplary advantages:
An imaging system with feedback-based micromirror configuration can increasing resolution in regions of interest, decrease resolution elsewhere in a FOV and improve image quality while maintaining the original FOV.
In a related advantage a first configuration of the micromirror array can uniformly spread the incoming FOV from a lens across a detector array. The array can generate first sensor data. A second configuration of the micromirror array can reconfigure a complex shaped plurality of the micromirrors to increase resolution in regions on interest and thereby generate second sensor data. Processing circuitry can use knowledge of the first and second configurations to combine the first and second data to generate a single image. The single image can comprise enhanced resolution in the regions of interest (e.g., at time of flight or color boundaries, around objects, faces, or intensity boundaries) from the second sensor data and background non-enhanced portions from the first sensor data. The micromirror mirror array can be reconfigured faster than a traditional zoom lens, thereby reducing motion distortion when combining first and second data.
In another advantage several embodiments provide for dynamically identifying a complex shaped region of interest (e.g., surrounding a vehicle) that can then be used to reconfigure a corresponding complex shaped subset of micromirrors. A complex shape region of interest can be a complex shape subset of a FOV and can include simple and complex curves or multiple sides (e.g., 5 or more distinct sides).
In another advantage various computer processing techniques can be used to identify regions of interest such as object classification, boundary detection, boundary extrapolation (e.g., predicting a location of some or all of a boundary), iterative boundary localization, facial recognition, location classification (e.g., urban, rural, or indoor). Computer processing techniques used to identify regions of interest from sensor data can use sensor fusion (e.g., combining multiple types of data), can prioritize or score regions of interest. In a related advantage computer processing can generate a profile or range of resolutions by reconfiguring a plurality of micromirrors. For example a region of interest can cause a subset of micromirrors to generate a resolution of 10 detector elements per square degree at the center of a region of interest in the FOV. The circuitry can further reconfigure a second subset of the micromirrors to generate lower resolution of 5 detector elements per square degree at the detector array for a portion of the region of interest surrounding the center of the region of interest.
In another advantage micromirror array can be iteratively reconfigured to progressively enhance resolution based on sensor data gathered from a previous iteration. Hence a micromirror array in a LIDAR could iteratively select regions of interest in which time of flight discrepancies indicate depth or range differences. After each iteration the detector array can generate sensor data indicating subsets of the previous regions of interest in which boundaries still require localization, thereby forming new regions of interest.
In another advantage, data-drive reconfiguration of the micromirror array enables a smaller photodetector array to perform like a more expensive, larger detector array. For example, consider an imaging system with a 100×100 degree FOV sensed with a 200×200 pixel or element photodetector array. The total angular area of the FOV is 100×100 or 10,000 square degrees. The total number of photodetector elements is 40000 and the average angular resolution is 4 pixels per square degree. An embodiment of the present disclosure can identify a region of interest with a complex shape (e.g., a hexagonal 2D shape with area of 100 square degrees in the FOV). The imaging system can then configure a micromirror array to increase the resolution to 100 pixels per square degree for a region of interest (e.g., equivalent to the average resolution of a 1000×1000 element photodetector). The imaging system can reduce the resolution to 3 pixels per square degree in the remainder of the FOV outside the region of interest, so as to sample the entire FOV. In this way the imaging system can sample the same 100×100 FOV while acting like a more expensive 1000×1000 element photodetector array in the region of interest.
In a related advantage the imaging system of the previous example can generate a smaller set of sensor data using anisotropic resolution and only increasing resolution in selected region(s) of interest.
Instead of generating a uniform laser pulse density throughout the FOV, the disclosed techniques provide for non-uniform laser pulse density by dynamically steering a laser based on data indicating the location of important features in the FOV (e.g., boundaries of an object, a person recognized in digital image). This data-driven non-uniform laser pulse spacing has the further benefit of further localizing the important features.
In another advantage the boundaries of objects in the FOV can be progressively localized by refining laser steering parameters in regions of the FOV. The disclosed techniques can improve speed detection for objects in the FOV. The accuracy of speed detection in a laser range finding scan is related to the ability to accurately determine the object boundary during each scan. The disclosed techniques can estimate the boundary location and dynamically steer the laser to investigate the boundary location.
The disclosed techniques enhance the speed of object classification, using boundary localization and dynamic laser pulse density selection.
With the advent of solid-state laser range finders with low azimuthal range (e.g. 90-120 degrees) the danger of high-intensity laser pulses is often confined to a threshold distance in a narrow range of angles. Aspects of the present disclosure provide improved accuracy and timeliness of detecting future intrusion into the path of high-intensity laser pulses. The disclosed laser range finder can improve laser safety by using eye-safe intensity guard pulses in dedicated strategically placed guard regions of a FOV to trigger intensity reduction in neighboring adaptive-intensity regions before an object has a chance to reach the adaptive-intensity region. In another advantage the disclosed systems can use low intensity laser pulses to discover objects, thereby maintaining compliance with safety requirements.
In a related area, a laser range finder can use machine learning to discover common intrusion paths into high intensity laser beams and can subsequently generate guard regions around these path, thereby making the high-intensity laser pulses contingent on analysis of common intrusion paths. In another advantage, the disclosed laser range finder can dynamically steer a laser beam to monitor guard regions first during a scan of the FOV before subsequently generating high intensity laser pulses.
Previous high-intensity laser systems must react quickly to objects to avoid damage caused by the laser intensity. The disclosed laser range finder provides increased reaction time using lower-intensity laser pulses to determine if an object is likely to intersect with high-intensity laser pulses, thereby reducing the number of false positive intensity reductions in the adaptive-intensity regions.
Embodiments of the present disclosure provide the further advantage of enabling analysis of the trajectory of objects in the guard region using lower intensity (e.g. eye-safe) laser pulses. In a related advantage the number of false positive intensity reductions is further reduced by using trajectory determination of objects in the guard region. In one embodiment, the trajectory of an object in the guard region can be safely measured using lower-intensity laser pulses and used to determine the intensity of laser pulses in the adaptive-intensity region. This is advantageous because as an autonomous vehicle with a laser range finder moves down an urban street the majority of pedestrians (e.g. on a sidewalk) enter the FOV at a far distance in the center of the FOV and proceed to move away to the edge as they approach the vehicle. This effect is similar to how stars in science fiction movies (e.g. Start Trek) or stars in video games (e.g. Galaga by NAMCO Inc.) tend to move from the center of the FOV to the sides due to the motion of the observing platform (e.g. the space ship). For this reason, as an autonomous vehicle moves the majority of pedestrians appear to move along a path from the middle of the FOV at far distances (e.g. 100 m) to the edge as they approach the autonomous vehicle. The disclosed embodiments provide a greater reaction time to determine if objects are moving in a typical manner and react accordingly.
In a related advantage, several embodiments provide for adapting the size, intensity and location of guard regions to adapt to different driving conditions. For example, a vehicle stopped at a crosswalk can implement wide guard regions with very low intensity, since the primary danger is a person walking in front of the vehicle. At high speeds guard regions can be narrowed and extended in range to protect people as the vehicle turn.
In digital photography light from is received at a sensor form many points in the local environment at once. In contrast, a laser range finder can use a relatively small number of lasers (e.g., 1-64) to generate laser pulses aimed sequentially at a number of points (e.g., 100,000) to perform laser ranging scans of the FOV. Hence, the laser pulses (e.g., and corresponding time of flight measurements in discrete directions) represent a scarce resource and the FOV is often undersampled with respect to sensing detailed boundaries of objects in the local environment. Many LIDARs mechanically rotate with a constant or nearly constant angular velocity. Such rotating LIDARs can sweep one or more lasers through a deterministic range of directions (e.g., each laser sweeping through a 360 degree azimuthal range at a fixed elevation angle). This type of operation does not constitute dynamically steering the laser(s) in a LIDAR. The angular momentum of the spinning portion in a mechanical LIDAR prevents rapid changes in angular velocity. Each laser in a mechanical LIDAR can generate a uniformly spaced sequence of laser pulses in a 1-D angular range. The angular velocity can be selected for many mechanical LIDAR (e.g., 5-20 Hz for the HDL-64E from Velodyne Inc. or Morgan Hill, Calif.), but remains constant from one rotation to the next.
A uniform scan of the entire FOV is simple and somewhat inherent in rotating LIDARS, but is sub-optimal for gathering the most information from the FOV. For example, large sections of the FOV (e.g., Walls and roads) can return a predictable, time invariant, homogeneous response. A modern LIDAR can scan over 2 million points per second. Hence one embodiment of the present technology tries to select the 2 million scan points with the most information (e.g., edges or boundaries) by steering the laser in a dynamic manner.
Recently, advancements in electronically-steerable lasers and phased array laser beam forming have made it possible to dynamically steer a laser within a FOV. A steerable laser can be mechanically-steerable (e.g., containing moving parts to redirect the laser) or electronically-steerable (e.g., containing an optical phased array to form a beam at in one of many directions). For the purpose of this disclosure a steerable laser is a laser assembly (e.g., including positioning components) that can change the trajectory or power level of a laser beam. For the purpose of this disclosure a steerable laser is dynamically steerable if it can respond to inputs (e.g., user commands) and thereby dynamically change the power or trajectory of the laser beam in the course of a scan of the FOV. For the purpose of this disclosure dynamically steering a laser is the process of providing input data (e.g., instructions such as laser steering parameters) to a steerable laser that causes the laser to dynamically modulate the power or trajectory of the laser beam during a scan of the FOV. For example, a laser assembly that is designed to raster scan a FOV with a constant scan rate (e.g., 10 degrees per second) and pulse rate (e.g., 10 pulses per second) is not being dynamically steered. In another example, the previous laser assembly can be dynamically steered by providing input signals and circuitry that dynamically changes the angular velocity of the laser assembly to generate non-uniformly spaced laser pulses in the FOV, based on the input signals (e.g., thereby generating an image on a surface in the FOV). A trajectory change can be a direction change (i.e., a direction formed by a plurality of pulses) or a speed change (i.e., how fast the laser is progressing in a single direction across the FOV). For example, dynamically changing the angular speed across a FOV of a pulsed laser with a constant direction causes the inter-pulse spacing to increase or decrease thereby generating dynamically defined laser pulse density.
In the context of the present disclosure most rotating LIDAR do not comprise dynamically steerable lasers since neither the power nor the trajectory of the laser beam is dynamically controllable within a single scan. However a rotating or mechanical LIDAR can be dynamically steered. For example, by providing input data that causes the laser to dynamically vary the laser pulse rate within a scan of the FOV, since the net result is a system that can guide or steer the laser to produce a non-uniform density laser pulse pattern in particular parts of the FOV.
Recently, electronically scanned LIDAR such as the model S3 from Quanergy Inc. of Sunnyvale, Calif. have been developed. These solid-state electronically scanned LIDAR comprise no moving parts. The absence of angular momentum associated with moving parts enables dynamic steering of one or more lasers in electronically scanned solid-state LIDAR systems.
In many laser range finding systems the laser is periodically pulsed and the exact pulse location in the FOV cannot be controlled. Nevertheless such a periodic pulse laser can be used with the present disclosure to produce a complex shaped region of higher pulse density than the area surrounding the region by increasing the laser dwell time within the region. In this way a periodically pulsed laser will produce a greater density of pulses in the complex shaped region of a FOV. For the purpose of this disclosure a complex shaped region is a region having a complex-shaped perimeter such as a perimeter with more than four straight edges or a perimeter with one or more curved portions and two or more distinct radii of curvature. Exemplary complex-shaped regions are, a region with a pentagonal perimeter, a hexagonal perimeter an elliptical perimeter or a perimeter capturing the detailed outline of a car. Other laser range finding systems transmit a continuous laser signal, and ranging is carried out by modulating and detecting changes in the intensity of the laser light. In continuous laser beam systems time of flight is directly proportional to the phase difference between the received and transmitted laser signals.
In one aspect the dynamically steered laser range finder can be used to investigate a FOV for boundaries associated with objects. For example, a small shift in the position of the LIDAR laser may identify a large change in TOF associated with the edge of an object 100 ft away. In contrast RADAR has much greater beam divergence and hence a much wider spot size impacts the object (often many times the object size). Hence the reflections from beam scanned RADAR represent the reflections from many points on the object, thereby making beam steered RADAR useful for object detection but impractical for performing detailed boundary localization. Hence, due in part to the large beam divergence of RADAR beams, a small change in radar beam direction can provide little if any actionable information regarding the edges of an object. In contrast the spot size of the laser remains small relative to the boundary of many important objects (people, dogs, curbs). The present technology can enable the boundaries (e.g., edges) of objects to be dynamically determined by a process of iteratively refining the scan points for the electronically steered LIDAR. For example, the LIDAR can use a bisection algorithm approach to iteratively search for the boundary of a pedestrian in the FOV. The LIDAR could first receive an indication that point P1 in a point cloud has a TOF consistent with the pedestrian and can scan iteratively to the right and left of P1 with decreasing angular range (e.g., in a bisection approach) to estimate the exact location of the boundary between the pedestrian and the surrounding environment. In general, this technique can be used to dynamically configure a laser in a LIDAR to investigate changes in TOF within a point cloud to iteratively improve boundary definition.
Unlike digital cameras where light is received form many points at once, a laser range finder can rely on a relatively small number of laser beams (e.g. 1-64) aimed sequentially at a number of points (e.g. 100,000) during each scan of the FOV. Hence, the measurement density of laser ranging systems is often much lower than digital cameras. The laser pulses represent a scarce resource and the FOV is often undersampled with respect to sensing detailed boundaries or changes in topology. For example, a tree in the field of view could be scanned with 1000 points during a scan of the FOV and the same tree could occupy one million pixels in a digital camera image. For the purpose of this disclosure the FOV of a laser transmitter is the set of all directions in which the laser transmitter can emit a laser light. For the purpose of this the FOV of a detector (e.g. a photodetector) is the set of all directions along which the detector can detect light (e.g. a laser pulse). The FOV of a laser range finder is set of all directions in which the laser range finder can perform laser range finding (e.g. the set of all directions in which the laser range finder can both transmit and receive laser light). For the purpose of this disclosure a single scan of a FOV by a laser range finder is the process of performing laser ranging measurements in the largest substantially unique set of directions (e.g. the longest sequence of directions that does not repeat or cover a substantially similar portion of the FOV). In a simple example, a rotating laser range finder may scan the FOV by performing a 360 degree revolution. A raster scanning laser range finder may scan he FOV by performing 10 left to right sweeps of a FOV and changing the elevation angle of the a laser generator after each sweep to cover the entire FOV.
LIDARs often provide laser ranging in a plurality of directions (e.g. a FOV) and thereby generate data for a 3D topology map of the surroundings. To accomplish this LIDAR can have a steerable laser assembly. For the purpose of this disclosure a steerable laser assembly is an assembly that scans one or more laser beam within a FOV. A steerable laser assembly can include a laser generator (e.g. a laser diode) and a laser positioner (e.g. a rotating scanning mirror) to position the laser beam in a variety of directions in during a scan of the FOV. The steerable laser assembly can be mechanically-steerable (e.g. containing moving parts to direct a laser beam) or electronically-steerable (e.g. containing an optical phased array to form a laser beam at in one of many directions).
Many LIDARs have a mechanically steerable laser assembly that rotates with a constant angular velocity and thereby scans the FOV with uniform measurement spacing (e.g. 1 laser pulse and 1 measurement for every 1 degree of the azimuthal FOV). The pattern of generated laser pulses is uniform and largely determined by the angular velocity of the rotating components. The angular velocity can be selected for many mechanical LIDAR (e.g. 5-20 Hz for the HDL-64E from Velodyne Inc. or Morgan Hill, Calif.), but remains constant (or nearly constant) from one rotation to the next. The uniform angular spacing of laser pulses within the FOV is simple and somewhat inherent in rotating LIDARs, but is sub-optimal for gathering the most information from the FOV. For example, large sections of the FOV can return a predictable, time-invariant, homogeneous response, such as reflections from walls or unoccupied sections of a highway.
In a mechanical LIDAR the inertia of the spinning components prevents rapid changes in the angular velocity that would be necessary to dynamically steer a laser beam to produce a complex non-uniform and dynamically defined patterns of laser pulses. Recently, advancements in electronically-steerable lasers and phased array laser beam forming have made it possible to dynamically steer a laser beam within a FOV. Electronically-scanned LIDAR are solid-state and comprise no moving parts (e.g. the model S3 from Quanergy Inc. of Sunnyvale, Calif.). In a solid state LIDAR, the absence of inertia associated with moving parts makes it possible to move a laser beam along a complex trajectory thereby producing a series of laser pulses with non-uniform spacing, density, and location in the FOV.
For the purpose of this disclosure, a dynamically steerable laser assemblies are a subset of steerable laser assemblies wherein the assembly can dynamically steer one or more laser beams by accepting inputs (e.g. user commands) and thereby dynamically change aspects of the laser beam such as beam power, spot size, intensity, pulse repetition frequency, beam divergence, scan rate or trajectory. A dynamically steerable laser assembly can change aspects of one or more laser beams several times during a scan of the FOV. For example, a differentiating aspect of many dynamically steerable laser assemblies over traditional laser assemblies is circuitry operable to process instructions while the laser beam scans the FOV and continually adjust the direction of a laser beam. This is similar to the dynamic manner in which a 3D printer dynamically rasters a polymer filament to print an arbitrary shaped object. A traditional mechanically steered LIDAR, with associated inertia, can only implement small changes in angular velocity during each scan (e.g. changing from 20 Hz to 20.5 Hz scan rate in the course of a single 360 degree rotation). In contrast, it can be appreciated that a dynamically steerable LIDAR can make several changes to aspects of the laser pulse pattern in the course of a single scan of the FOV (e.g. rapidly changing the trajectory of a laser beam by 90 degrees within 10 milliseconds or tracing the outline of a complex shape with many turns during a single scan).
For the purpose of this disclosure, dynamically steering a laser beam with a steerable laser assembly is a process of providing input data to the steerable laser assembly that causes the steerable laser assembly to dynamically modulate at least one aspect of the resulting laser pulse sequence during a scan of the FOV. Exemplary modulated aspects can include the beam or pulse power, spot-size, intensity, pulse repetition frequency (PRF), beam divergence, scan rate or trajectory of the laser beam. For example, a laser assembly that is designed to raster scan a FOV with a constant scan rate and pulse rate (e.g. PRF) is acting as a steerable laser assembly but is not being dynamically steered. The distinction is that such a laser assembly is not receiving input or acting on previous input and dynamically altering aspects of the beam pattern during the course of each scan of the FOV. However, the same steerable laser assembly could be dynamically steered by providing input signals that cause the steerable laser assembly to generate a variable laser power at locations in the FOV, based on the input signals (e.g. thereby generating an image on a surface in the FOV). A trajectory change can be a direction change (i.e. a direction formed by a plurality of pulses) or a speed or scan rate change (i.e. how fast the laser is progressing in a single direction across the FOV). For example, dynamically steering a steerable laser assembly can be dynamically changing the angular velocity, thereby causes the inter-pulse spacing to increase or decrease and generating a dynamically laser pulse density. In one aspect, dynamic steering can often be recognized as the process of implementing dynamic control of a laser pulse pattern during a scan of a FOV.
In the context of the present disclosure, many rotating LIDAR do comprise steerable laser assemblies, but these assemblies are not dynamically steerable since neither the power nor the trajectory of the laser beam is dynamically controllable within a single scan of the FOV. However, a rotating or mechanical LIDAR could be dynamically steered, for example, by providing input data that causes the laser to dynamically vary the laser pulse rate within a scan of the FOV, since the net result is a system that can guide or steer the laser to produce a non-uniform density laser pulse pattern in particular parts of the FOV.
In many laser range finders the laser is periodically pulsed as the laser assembly moves along a trajectory and the exact location of each laser pulse in the FOV is controlled. Nevertheless such a periodically pulses laser generator can be used in a steerable laser assembly to produce a complex shaped region with greater than average spatial density pulse density, For example, by increasing the laser dwell time within the complex shaped region. In this way, a periodically pulsed laser generator (e.g. a laser diode) can produce a greater density of pulses in the complex shaped region. Other laser range finding systems transmit a continuous laser signal, and ranging is carried out by modulating and detecting changes in the intensity of the laser light. In a continuous laser beam systems the distance to a reflection location can be determined based on the phase difference between the received and transmitted laser signals.
In one aspect, a dynamically steered laser range finder can be used to mine the FOV for the boundaries. For example, a LIDAR can generate laser pulses with a 3 milliradian beam divergence, thereby resulting in a 2 cm by 2 cm laser spot size at a distance of 200 m. This small laser spot size enables the LIDAR to identify the boundaries of an object at 200 m. In many cases the resolution of objects at considerable range is limited by the number of pulses devoted to an object rather than the ability of each pulse to identify a boundary. Therefore, once a boundary is detected a dynamically steerable laser assembly could be dynamically steered to investigate and refine estimates of the boundary by devoting more pulses to the object. In contrast, RADAR has much greater beam divergence and hence a much wider spot size impacts the object (often many times the object size). Hence, the reflections from beam-steered RADAR represent the reflections from many points on the object, thereby making beam steered RADAR useful for object detection but impractical for detailed boundary determination or localization. Hence, in a RADAR a small change in beam angle provides little if any actionable information regarding the edges of an object. In contrast the spot size of the laser remains small relative to the boundary of many important objects (people, dogs, curbs). The present technology enables the boundaries of such objects to be dynamically determined by a process of iteratively refining the scan points for the electronically steered LIDAR. For example, a LIDAR with dynamic steering could use a bisection algorithm approach to iteratively search for the boundary of a pedestrian in the FOV. The LIDAR could first process laser reflection data to identify that a 3D point P1 in the point cloud has a TOF consistent with the pedestrian and can subsequently scan iteratively to the right and left of P1 with decreasing angular range (e.g. in a bisection approach) to estimate the exact location of the boundary between the pedestrian and the surrounding environment. In general, this technique can be used to investigate changes in range (e.g. time of flight changes) within a point cloud to iteratively improve boundary definition or boundary location estimates.
In contrast
In
Steerable laser assembly 120 can comprise one or more laser generators 420, a laser positioner 430, and one or more detectors 440. The one or more laser generators 420 can be laser diodes (to produce one or more laser beams (e.g., beam 435) at one or more locations in the FOV determined by the laser positioner 430. Laser positioner 430 functions to steer one or more laser beams (e.g., beam 435) in the FOV based on the laser steering parameters. Laser positioner 430 can mechanically steer a laser beam from laser generator 420. Rotating LIDARs often use a mechanically steered laser positioner. An exemplary mechanically steered laser positioner 430 can include mechanical means such as a stepper motor or an induction motor to move optical components relative to the one or more laser generators. The optical components in an exemplary mechanical laser positioner can include one or more mirrors, gimbals, prisms, lenses and diffraction grating. Acoustic and thermal means have also been used to control the position of the optical elements in the laser positioner 430 relative to the one or more laser generators 420. Laser positioner 430 can also be a solid state laser positioner, having no moving parts and instead steering an incoming laser beam using electronic means to steer the laser beam 435 in an output direction within the FOV. For example, an electronically steerable laser assembly can have a solid state laser positioner comprising a plurality of optical splitters (e.g., Y-branches, directional couplers, or multimode interference couplers) to split an incoming laser beam into multiple portions. The portions of the incoming laser beam can then be transmitted to a plurality of delay line where each portion is delayed by a selectable amount (e.g., delaying a portion by a fraction of a wavelength). Alternatively, the delay lines can provide wavelength tuning (e.g., selecting slightly different wavelengths from an incoming laser beam). The variable delayed portions of the incoming laser beam can be combined to form an output laser beam at an angle defined at least in part by the pattern of delays imparted by the plurality of delay lines. The actuation mechanism of the plurality of delay lines can be thermo-optic actuation, electro-optic actuation, electro-absorption actuation, magneto-optic actuation or liquid crystal actuation. Laser positioner 430 and one or more laser generators 420 can be combined onto a chip-scale optical scanning system such as DARPA's Short-range Wide-field-of-view extremely agile electronically steered Photonic Emitter (SWEEPER).
Detector 440 can contain light sensors 450 (e.g., photodiodes, avalanche photodiodes, PIN diodes or charge coupled devices CCDs), signal amplifiers (e.g., operational amplifiers or transconductance amplifiers), a time of flight calculator circuit 455 and an intensity calculator 460. Detector 440 can comprise one or more photodiodes, avalanche photodiode arrays, charge coupled device (CCD) arrays, single photon avalanche detectors (SPADs), streak cameras, amplifiers and lenses to focus and detect reflected laser light from laser beam 435. The construction of the steerable laser assembly 120 can co-locate detector 440 and laser positioner 430 such that detector 440 is pointed in the direction of the outgoing laser beam and can focus the detector on a narrow part of the FOV where the reflected light is anticipated to come from.
The steerable laser assembly 120 of laser range finder 405 can generate a pulsed or continuous laser beam 435. Steerable laser assembly 120 can receive one or more laser reflections 445 corresponding to laser beam 435. Laser range finder 405 can contain a light sensor 450 to detect reflected light from the laser pulses or continuous laser beam.
Steerable laser assembly 120 can contain a time of flight calculator 455 to calculate the time of flight associated with a laser pulse striking an object and returning. The time of flight calculator 455 can also function to compare the phase angle of the reflected laser beam with the phase of the corresponding outgoing laser beam and thereby estimate the time-of-flight. Time of flight calculator 455 can also contain an analog-to-digital converter to detect an analog signal resulting from reflected photons and convert it to a digital signal. Laser range finder 405 can contain an intensity calculator 460 to calculate the intensity of reflected light.
Laser range finder 405 can contain a data aggregator 465 to gather digitized data from time of flight calculator 455 and intensity calculator 460 or 3D location calculator 464. Data aggregator 465 can group data into packets for transmitter 470 or sensor data processor 475. Laser range finder 405 can contain a transmitter 470 to transmit data packets. Transmitter 470 can send the data to a processing subassembly (e.g., a computer or a remote located sensor data processor) for further analysis using a variety of wired or wireless protocols such as Ethernet, RS232 or 802.11.
Laser range finder 405 can contain a sensor data processor 475 to process sensor data and thereby identify features or classifications for some or all of the FOV. For example, data processor 475 can identify features in the FOV such as boundaries and edges of objects using feature identifier 480. Data processor 475 can use feature localizer 485 to determine a region in which the boundaries or edges lie. Similarly a classifier 490 can use patterns of sensor data to determine a classification for an object in the FOV. For example, classifier 490 can use a database of previous objects and characteristic features stored in object memory 495 to classify parts of the data from the reflected pulses as coming from vehicles, pedestrians or buildings. In the embodiment of
When operable linked to steerable laser assembly 120 the processing subassembly 520 can perform one or more embodiments of the method to find, utilize and correct for a remote mirror in the FOV of laser range finder 510.
A laser steering parameter can be a region width 504 or a region height 506. The width and height can be expressed in degrees within the FOV. One exemplary set of laser steering parameters could include a start location, region width and region height thereby defining a four sided region in the FOV. Other laser steering parameters in the exemplary set of laser steering parameters can indicate how to tailor a scan within this region, such as laser scan speed 514, laser pulse size 516 or number of laser pulses 518.
A laser steering parameter can be one or more region boundaries 508 defining the bounds of a region. A laser steering parameter can be one or more laser pulse locations 511. Pulse locations 511 can provide instructions to a steerable laser to move to corresponding positions in the FOV and generate on or more laser pulses. In some embodiments the laser can be generating a laser beam while being steered from one location to another and can dwell for some time at the laser pulse locations. In other embodiments the steerable laser can use these points 511 to generate discrete pulses at defined locations. In such embodiments the laser beam can be generated at discrete pulse locations and can dwell at the pulse location for some time.
A laser steering parameter can be one or more path waypoints 512, which define points in a FOV where a steerable laser can traverse or points at which the steerable laser can implement direction changes.
Turning to
Turning to
Turning to
The direction of each of the intervening pulses 810a-e is indicated by the 2-D location in the FOV 610. The direction of intervening pulse 810a can be based one or more of the directions of the corresponding pair of laser pulses 725a. For example, path 820 can be designed to place pulse 810a midway between the laser pulses in pair 725a. Path 820 can place intervening pulses 810a-e at specified angular direction relative to one of the pulses in each of the pairs of laser pulses with TOF difference. For example, the first sequence of laser pulses produced by steering the LIDAR 605 along path 625 in
Intervening laser pulses (e.g., pulses 810a-b) can be added to the sequence of laser pulses. In one aspect intervening laser pulse 810a causes laser pulse pair 725a in
Turning to
In
In
In several embodiments, a LIDAR can apply a boundary localization test to each point in an existing set of laser pulses with corresponding directions and TOF values. The localization test can define several angular ranges. Consider that laser reflection 710 in
Several embodiments of
Various embodiments provide for calculating a confidence value or standard deviation associated with the direction (i.e., the angular offset to reach a new search zone defined by an estimated boundary location or vector). For example, everyday objects can have boundaries or edges with simple shapes (straight lines or simple curves) arranged in a direction relative to an observation point. Hence while it may be impractical for a rotating LIDAR to try to dynamically track and scan the boundary of object at an arbitrary orientation, it may be more practical to use a dynamically steerable LIDAR. In comparison to a steerable RADAR that tracks an objects movement from one scan to another and can predict a direction for the object, the disclosed PBL method can estimate the edges of an object within a single scan by finding a first portion of an edge and predict a direction for the edge (based on curve fitting, object classification or extrapolation). The method can then scan a laser beam in a pattern at a second location some distance along the predicted direction of the boundary in the FOV. Turning to
In a related embodiment, a LIDAR can scan a path including a sequence of orientations in a first 2-D search region 1160 of a FOV. While scanning the path, the LIDAR can generate a plurality of laser pulses, receive a corresponding sequence of laser reflections and calculate a TOF corresponding to each of the outgoing laser pulses. The LIDAR can identify the presence of a TOF boundary (e.g., the edge of a vehicle or the edge 1111 of a roadway), by identifying one or more nearest neighbor pairs of laser reflections for which the TOF difference is greater than a TOF threshold. The LIDAR can calculate a set of boundary locations (e.g., locations 1162a and 1162b) based on the TOF measurements from the first search region 1160. The LIDAR can process one or more locations in the set of boundary locations (e.g., locations 1162a and 1162b) to predict an estimated boundary location 1163a, located outside the first search region. The LIDAR can generate a set of laser steering parameters, based on the estimated boundary location and dynamically steer a laser 1106 based on the laser steering parameters to generate a second plurality of laser pulses (e.g., including laser pulse 1170) in a second search region. In this way a LIDAR scan can be guided by identifying and adding directions in a FOV (e.g., locations in a FOV) that lie on a TOF boundary, predicting and estimated boundary location outside a first search region and scanning a second search regions with laser pulses based on the predicted trajectory of the TOF boundary. The method can be performed iteratively in the course of a single scan by building up a set of confirmed boundary locations, predicting estimated boundary locations and scanning a second search region around the estimated boundary location. In one embodiment of an EPBL method illustrate in
In another embodiment of a EPBL method a LIDAR 1105 can track several TOF boundaries 1110 and 1111 simultaneously, by several distinct sets of boundary locations and periodically generating a new search regions for each based on a new extrapolated estimated boundary location. An EPBL method that tracks several boundaries at once can perform different functions in parallel such as extrapolating an estimated boundary location for a first boundary while scanning a new search region for a second boundary. Similarly an EPBL method can perform a wide angle 2-D scan of a FOV to search for new TOF boundaries while extrapolating boundary locations and tracking one or more previously discovered boundaries.
Turning in detail to
The micromirror array 1310 can be used to dynamically select inputs for the FOV 1325 of detector 1315. Micromirror array 1310 can occupy the entire FOV 1325 of a detector or photodetector array 1315. In various configurations the micromirror can then present to the detector 1315 light reflections from one of multiple CFOB s, light reflection multiple CFOB s simultaneously with light reflections from each CFOB directed to different parts of the detector. Alternatively, micromirror 1310 can then present to the detector 1315 light reflections from multiple CFOBs simultaneously with light from each CFOB directed to overlapping parts of the detector. Mirrors (e.g., 1320) in some or all of the micromirror arrays can be arranged at different angles to form angled reflectors to focus light reflections from all or portions of a CFOB onto a single detector element or a few detector elements. This can be useful for detecting if any optical fiber in a portion of the output surface of a CFOB is carrying a light reflection. Alternatively micromirrors can form a convex mirror arrangement, thereby spreading light reflections from a portion of the CFOB output surface over a wider portion of the detector (e.g., a wider range of elements in a detector array). In this way the micromirror array can magnify, combine, select and overlap portions of one or multiple CFOBs onto a photodetector 1315. The usefulness of the micromirror array is enhances by available light reflections from multiple FOVs based on the plurality of CFOBs.
Lidar with a Micromirror Array for Dynamic Reflection Distribution
In a related group of embodiments, a flash LIDAR can use a micromirror array to dynamically select one or more subsets of a FOV to transmit to a detector or detector array, and thereby improve the LIDAR resolution. While 2D digital cameras and 3D time-of-flight cameras are similar in some aspects, the different objectives makes scaling detector array in LIDARs challenging. Specifically, 2D digital cameras integrate the charge (photon current) at each pixel on the CCD array over a relatively large acquisition time (e.g., 10-100 milliseconds) often with little regard for when photons arrive within the acquisition time window. Subsequently, a readout circuit can read the charge stored on many pixels in a serial or parallel manner. Advances in the speed of readout circuitry have enables the resolution of 2D cameras (e.g., number of pixels) to outpace the complexity of the corresponding readout circuitry. For example, readout circuits in 2D cameras are servicing increasing numbers of pixels per readout circuit, thereby enabling higher resolution 2D digital camera. Conversely, 3D time-of-flight cameras are designed to determine when light reflection arrives at the detector array and thereby determine distance to a reflection source. Each pixel often has associated electronics (e.g., transimpedance amplifiers, phase comparators or timing circuits). Hence LIDAR resolution (numbers of pixels per array) has lagged behind that of 2D digital cameras and ways to increase this resolution remain a challenge.
In one aspect, reflection positioner circuitry 1330 can function to adjust the 488 micromirrors in each of the portions 1450a and 1450b to focus light reflections from the corresponding portions of the micromirror FOV onto corresponding detector elements 1460a and 1460b respectively. For example, reflection positioner circuitry 1330 can instruct the 488 micromirrors in portion 1450a to form a concave reflector with a focal distance equal to the detector array. This can provide operation similar to direct illumination of the detector element by laser reflections from a portion of the FOV. This mode can be useful for detecting weak reflections, since many micromirrors can combine laser reflections from a single part of the FOV (e.g., a 0.5×0.5 degree portion corresponding to 488 micromirrors).
In one aspect, the 3D location calculator 464 can also receive data indicative of the configuration of the micromirror array 1310. For each light reflection in the set of light reflections the 3D location calculator can generate a 3D location indicative of a reflection location corresponding to the light reflection. The 3D location can be based on a detector element (e.g., the position in a detector array where the reflection was sensed) and further based on the configuration of the micromirror array (i.e., the subset of directions in the FOV being deflected towards the detector array). For example, a detected light reflection at detector element 1460a can indicate a reflection at a location encompasses by region 1430a in the FOV 1420. The micromirror array configuration can further refine the portion of the FOV to indicate the reflection came from the upper left portion 1435 of region 1430a. The time-of-flight between the corresponding emitted light pulse and a light reflection can indicate the range to the reflection location within region 1435. Hence the various micromirror array configurations enable more unique 2D locations (i.e., 2D reflection directions) to be generated (i.e., measured) in a corresponding 3D point cloud, than the number of photodetector elements in array 1405. For example the configuration of
In one aspect, while the ratio of solid angle in FOV 1510 to micromirrors in the micromirror array can be fixed, the micromirror array can be dynamically configured (e.g., using reflection positioner circuitry 1330) to distribute the reflected laser beams in a dynamic manner. For example, reflected laser beams from region 1530 of FOV 1510 can be spread across region 1555 (comprising 4 pixels) of detector array 1525. Conversely, reflected laser beams from region 1540 are focused by region 1550 of the micromirror array on a single pixel 1560. In a similar way laser reflections from a subset 1575 of the micromirrors can be directed to a particular receiver element (e.g., pixel). In one embodiment, dynamically configuring micromirror array 1520 to spread laser reflection from a region 1530 across an increased number of receiver pixels can identify a time-of-flight (TOF) boundary (e.g., the edge of an object) in the FOV. For example sub-region 1570 of region 1530 can indicate a TOF boundary relative to the remainder of region 1530 and the TOF boundary can be identifies based in part on focusing subset 1575 of the micromirrors onto a dedicated group of pixels 1565 in detector array 1525 (i.e., across a wider angular range in the receiver array). LIDAR 1500 can iteratively localize a boundary by iteratively spreading a sub-region (e.g., 1570) identified to contain a TOF boundary across a greater portion of the receiver array (e.g., upon identification that region 1570 contains a TOF boundary, reconfiguring the micromirror array 1520 to focus a corresponding subset 1575 onto region 1565 or photodetector array 1525.
Micromirror array 1520 can be dynamically configured to increase or decrease the ratio of input solid angle from the FOV to output solid angle at the photodetector array based on variety of parameters such as scene classification (e.g., urban, suburban, or highway), the presence of a particular object (e.g., cars, people etc.) the presence of boundaries (e.g., a roadside, overpass or person outline). Micromirror array 1520 can also be configured to periodically enhance a sequence of regions in the FOV (e.g., to periodically enhance each portion of the FOV), thereby providing periodic resolution enhancement to one, some or all regions of the FOV.
In a related embodiment to LIDAR 1500 a digital camera can have a similar arrangement. Instead of a laser transmitter the digital camera can generate light or rely on ambient light. The digital camera can identify edges within the FOV (e.g., based on initial data received at a CCD array similar to receiver 1525). Upon identification of boundaries or edges in initial image data the digital camera can reconfigure a micromirror array to dynamically enhance boundary localization by spreading the boundary containing regions across more pixels in the receiver array. The output image can be a combination of data including uniform and non-uniform configurations of the micromirrors.
In one aspect a micromirror array can act like an electronically controllable transfer function for light, between an input lens of a camera and a photodetector array. For example, an analog micromirror array can perform a zoom function by deflecting a small portion of available FOV onto the photodetector array while simultaneously spreading the small portion over the detector. This has the effect of increasing image resolution (e.g., pixels per square degree of the field of view). However zooming in a portion of the FOV with the micromirror array can have the drawback of narrowing the FOV (i.e., zooming in on the scene). There are many applications where both enhanced resolution and a wide FOV are desirable. In one embodiment a method performed by an imaging system comprises providing at an aperture a 2D field of view (FOV) from a scene to a micromirror array having a first configuration, and thereby deflecting light with the micromirror array from the FOV onto a photodetector array. The method further comprises detecting with the photodetector array a first set of light measurements that span the FOV, processing the first set of light measurements and thereby identifying a region of interest (e.g., a portion of the FOV or scene containing an object edge or a face), in the FOV, having a first resolution at the detector array. The method further comprises configuring the micromirror array based at least in part on the identified region of interest and thereby detecting with the photodetector array a second set of light measurements spanning the FOV with a second resolution in the region of interest that is greater than the first resolution.
In one aspect the method can conserve the size (e.g., angular range) of the original FOV, thereby keeping people and pets in the frame and not distracting a user with an unwanted zoom effect. In another aspect the method can enhance image resolution while simultaneously conserving the original FOV; by configuring the micromirror array to compress light rays from one or more uninteresting portions of the FOV onto fewer pixels in the photodetector array (e.g., based on the first set of light measurements) and thereby enabling light rays from the region(s) of interest to be spread over more pixels to enhance the resolution. Therefore, by creating areas of sparse and denser light rays on the photodetector array simultaneously, the original FOV can be conserved.
In a system embodiment a processing subassembly with access to data from the photodetector array and micromirror configuration can correct for the distortive effect of the dense and sparse zones on the photodetector array and generate an eye-pleasing output image. In another embodiment, data from sensors or sources other than the photodetector array can be used to identify the region(s) of interest. In a second embodiment a method performed by an imaging system comprises: Processing sensor data indicative of a scene in the vicinity of a micromirror array and thereby identifying a region of interest in the sensor data, wherein the micromirror array has a field of view encompassing at least some of the scene, wherein the micromirror array comprises a plurality of micromirrors with an initial configuration that deflects light from the region of interest towards a detector array and thereby provides a first resolution at the detector array for the light from the region of interest, configuring the plurality of micromirrors in the micromirror array, based at least in part on the identified region of interest and thereby providing at the detector array a second resolution for light form the region of interest that is greater than the first resolution.
In a third embodiment the micromirror array can be part of a ranging subassembly for a light detection and ranging system (LIDAR). For example a flash LIDAR can illuminate a field of view (FOV) with flashes of light and gather reflections from the FOV at a photodetector array. A micromirror array can be configured based on an identified region of interest to non-uniformly spread the light reflections from the flashes of light based on the identified region of interest.
Turning to
In
In one aspect the high resolution portion of the detector array can have a high resolution based on the total available number of detector elements or pixels in the detector array, based on the size of the region of interest (e.g., the solid angle or area of the field of view identified as a region of interest based on the sensor data). For example, 25% of a 1000×1000 pixel detector array can be devoted to resolution enhancement. If a small region of interest (e.g., 10×10 square degrees around a face in the background) is identified in a FOV the micromirror array can be reconfigured to provide a very high resolution of 2,500 pixels per square degree. Alternatively if a larger region of interest (e.g., a 1000 square degree complex shaped region around the boundary of a vehicle) is identified the micromirror array can be reconfigured to provide a high resolution of 250 pixels per square degree. In both cases the total number of pixels devoted to resolution enhancement can be 250,000 or 25% of the total detector array.
In one embodiment a method comprises the steps of firstly obtaining a micromirror array, comprising micromirrors in a first configuration; secondly deflecting with the micromirror array a first set of light beams from a FOV towards a detector array; thirdly detecting with the detector array the first set of light beams and thereby generating first sensor data; wherein a subset of the first set of light beams are from a region of interest in the FOV and have a first resolution at the detector array; fourthly in response to processing the first sensor data, reconfiguring at least some of the micromirrors; and fifthly deflecting, with the at least some of the micromirrors, a second set of light beams from the region of interest to the detector array; wherein the reconfiguration of the at least some of the light beams causes the second set of light pulses to have a second resolution at the detector array greater than the first resolution.
In one aspect the reflection positioner 1330 can receive a set of instructions to reconfigure the micromirror array and thereby implement a transfer function between a light rays from a FOV and their placement and resolution on a photodetector array (e.g., FOV 1670 of imaging system 1600). The transfer function can aim to enhance resolution of regions of interest in the FOV such as boundaries, objects of interest, or new objects in need of classification. This dynamically implemented transfer function creates dynamically defined relationship between light rays from the local environment and the sensor data measured by the detector array. With the micromirror array in a configuration to enhance resolution of region(s) of interest the corresponding high-resolution sensor data gathered at the detector array is effectively distorted by the non-uniform configuration of the micromirror array. Hence in one aspect the knowledge of the transfer function by the reflection positioner 1330 can be used by a sensor data processor 475 to process the high-resolution sensor data to enable it to be combined or displayed with other sensor data from other configurations. Sensor data from the detector array can be decoded using knowledge of the micromirror array configuration to place the sensor data in a common frame of reference (e.g., a 2D or 3D array forming an image).
In another embodiment a reflection positioner can generate a set of positioning instructions operable to configure the micromirror array. The positioning instructions can generate a high-resolution region within the micromirror array that functions to deflect light from the FOV with a higher than average resolution or a higher than original resolution towards a corresponding high-resolution portion or region of the detector array. The high resolution region of the micromirror array can deflect light from a region of interest. For example the high-resolution region can have the shape of a line that captures the outline of an object (e.g., a car) in the local environment. The high-resolution region of the detector array can generate high-resolution data. The high resolution data can be processed according to a transfer function indicating the configuration of the micromirror array. This processing of the high-resolution data can place high-resolution data in a common frame of reference or to account for the magnifying effect of the high-resolution region of the micromirror array. The sensor data processor 475 can combine sensor data at a uniform or average resolution (e.g., used to generate the positioning instructions) with high-resolution data to form a 2D or 3D image 1860. For example an imaging system can gradually configure a micromirror array by iteratively processing sensor data, configuring regions of the micromirror array and gradually refining the resolution of regions of interest at the detector array. A 2D or 3D image can be formed by the sensing data from the detector array with the micromirror in the final configuration. Alternatively the 2D or 3D image can combine sets of sensor data from a plurality of configurations leading to a final configuration. For example an initial uniform configuration of the micromirror can serve to provide a foundation of sensor data. Subsequent configurations can provide additional sets of high-resolution sensor data from subsets of the whole FOV that when combined with the first sensor data set provide an enhanced resolution image of all of the FOV with enhanced resolution in dynamically defined regions of interest. For example imaging system 1600 can generate a 2D image or a 3D point cloud comprising sensor data from a first uniform scan of the FOV and a subsequent adaptive resolution scan based on processing data from the first uniform scan.
In one aspect a region of interest, high-resolution region of a micromirror array or a high resolution region of a detector array can be selected based on sensed object, a classification of an object
In a LIDAR embodiment a method comprises firstly generating with one or more emitters an outgoing set of light pulses; secondly deflecting with a micromirror array, having a field of view, a first set of light reflections corresponding to the outgoing set of light pulses; thirdly detecting at a detector array the first set of light reflections and thereby generating a first set of reflection data; fourthly processing the first set of reflection data and thereby identifying a location estimate for a region of interest in the FOV, wherein the region of interest has a first resolution at the detector; fifthly configuring the micromirror array based at least in part on the location estimate for the region of interest and thereby generating a second resolution at the detector for the region of interest that is greater than the first resolution.
Lidar with Direction Feedback
Turning to
Turning in detail to
Similarly, control circuitry can function to adjust the OPA to provide maximal intensity in the calibration direction when a corresponding input calibration signal 1975 commands the OPA to point in the calibration direction 1945. In one embodiment control circuit 1970 can assert a malfunction indicator signal 1985 (e.g., a 0-12V value) if, in response to the input calibration signal 1975 the OPA does orient the laser beam in the calibration direction 1945. The malfunction indication signal 1985 can connect the control circuit or the laser detector 1965 to a malfunction indicator pin 1990 on the enclosure 1902 of LIDAR 1900. In one embodiment both the input calibration signals 1975 and the offset adjustment signal can be generated by the control circuitry 1970.
In one embodiment A LIDAR comprises one or more emitters to generate a set of laser pulses, wherein each of the plurality of laser pulses has a corresponding direction and beam cross-section; a selective light modulator positioned in the path of the plurality of laser pulses, comprising a plurality of segments with electronically controllable transparency, and control circuitry operable coupled to the selective light modulator and configured to control for each of the plurality of pulses at the electronically controllable transparency of at least some of the plurality of segments to block laser light from at least some the corresponding beam cross-section of the each laser pulse and transmit at least some of the each laser pulse with a transmitted beam cross-section smaller than the corresponding beam cross-section.
Turning to
In a related embodiment a vehicle based laser range finding system 2170 can comprise a receiver to receive roadside range data and roadside LIDAR location data, a processor to transform the Roadside range data to a common origin (e.g., reference point) relative to onboard range data, wherein the transform is based on the roadside LIDAR location information and the vehicle location and finally combine the transformed roadside range data with onboard range data. The transformed roadside range data and the onboard range data can be combined in a single 3D point cloud.
Laser range finder 2720 is designed to address several challenges associated with safely generating a set of high-intensity laser pulses. One challenge is to diminish laser intensity and thereby eliminate the keepout zone 2758 before a person 2780 reaches the keepout zone. A related challenge is to increase the accuracy of indications of future ingress into a keepout zone, thereby decreasing the number of false positive ingress indications. For example, the challenge of false positive ingress indications can be to differentiate person 2780 on a trajectory that intersects the keepout zone from person 2770 who is in the vicinity of the vehicle 2710 but not in imminent danger of entering the keepout zone. Similarly person 2760 who is adjacent to the keepout zone (or perhaps at a distance beyond the keepout zone) but has a trajectory that will pass to one side of the keepout zone as vehicle 2710 moves down street 2715.
Previous solutions were to monitor for objects in the keepout-zone and discontinue laser pulses upon detection of a person. A disadvantage of this approach is that person 2780 is irradiated with high-intensity laser pulses for as long as it takes laser range finder 2720 to discover the presence of person 2780.
Turning in detail to the embodiment of
Laser range finder 2720 further generates a guard set of laser pulses (e.g. pulses 2750), each with an intensity below the threshold intensity in two guard zones 2740a and 2740b. The guard zones 2740a and 2740b are positioned on either side of the high-intensity zone, thereby providing that a large number of potential ingress trajectories (e.g. trajectory 2759) into the keep-out zone require an object to first travel through a guard zone. Laser range finder 2720 can contain a detector and a processing subassembly (e.g. processing subassembly 520 and detector 440 in
In several aspects the guard laser pulses and guard zones can provide sufficient time to analyze objects for potential future ingress into a high-intensity zone. This is useful because many objects can naturally move in a trajectory away from the high-intensity regions during monitoring the in guard zone. The guard zones can be sized to provide sufficient reaction time to determine aspects (e.g. trajectory) of objects. In one aspect, as vehicle 2710 drives down street 2715 person 2760 may appear in guard region. Person 2760 can be standing on a footpath beside street 2715. The guard region and associated reflection data can provide basis to determine the person 2760 is proceeding towards the right side of guard region 2740b, and hence is not on a collision course with keep-out zone 2758. In another aspect, a processing subassembly in laser range finder 2720 can process reflection data from the guard regions and identify that person 2780 is on a collision course with the keepout region. In one aspect a guard zone can be a region of space, adjoining a high-intensity zone, through which guard laser pulses travel, such that reflections from the guard laser pulses are operable to control the intensity of laser pulses in the adjoining high-intensity zone. Guard zones can be defined as the volume of space in which guard laser pulses are operable to provide reflections that can control at least in part the intensity of subsequent laser pulses in a high-intensity zone. In the embodiment of
In
Turning to
In
High-intensity zone 2730c in front of vehicle 2710 can also be protected by a guard zone 2740d that is dependent on the speed of the vehicle. For example, consider laser range finder 2720 generating high-intensity laser pulses in zone 2730c while traveling at 60 MPH on vehicle 2710. The high-intensity laser pulses can remain above a threshold intensity out to a threshold distance from laser range finder 2720, thereby generating keepout zone 2758 within the high-intensity zone 2730c. The probability of lateral intrusion into keepout-zone 2758 changes with vehicle speed. In many cases to probability of intrusion is small because vehicle 2710 would likely strike objects in the keepout zone 2758 at 60 MPH. Hence the angular range of forward facing guard regions can decrease as vehicle speed increases.
In the embodiments of
Laser reflections from the second flash can be used to determine the intensity or angular range for a third laser flash in zone 1150 in
For example, if two people own the same model of autonomous vehicle using embodiments of the present adaptive intensity laser range finder 21210, processing subassembly 520 can generate guard regions based on previous data (e.g. intrusion paths into high-intensity laser pulses) to best meet the goals of laser safety and ranging performance. Consider that a first driver may drive primarily in rural area with tree-lined streets and processing subassembly 520 can adapt to provide narrow guard regions or mask regions around the adaptive-intensity regions, thereby reducing false positive intensity reduction in the adaptive-intensity region caused by laser reflections form the trees. A second driver with the same model vehicle may drive primarily in urban areas where pedestrians often cross at cross-walks in front of the FOV. Processing subassembly 520 can adapt the guard regions to be wide and have a sufficiently low laser intensity (e.g. 1 mW/cm2) to remain eye-safe. In both bases the guard regions are comprises of laser pulses each with an intensity below a threshold intensity and control the intensity of laser pulses in a high-intensity portion of the FOV. In another aspect an autonomous vehicle (e.g. vehicle 2710) with a laser range finder 1210 according to the present disclosure can record intrusion events into an adaptive-intensity region of the FOV (i.e. where an intrusion into an active keepout zone occurred e.g. keepout zone 2758 in
At step 21304 a steerable laser assembly in a laser range finder steers at least one laser beam and thereby generates, a preliminary set of laser pulses in a guard region of the field of view, each with an intensity below a threshold intensity. At step 21306 detector in the steerable laser assembly detects a preliminary set of laser reflections corresponding to the preliminary set of laser pulses and thereby generating first reflection data. The first reflection data can indicate the direction and range corresponding to laser reflections in the set of laser reflections.
At step 21308 the laser range finder performs a safety test using the first reflection data and thereby generates a first result. In response to the first result, the steerable laser range finder steers at least one laser beam and thereby generates a first set of laser pulses in an adaptive-intensity region of the field of view, each with an intensity above the threshold intensity. At step 21320 the steerable laser assembly generates, a guard set of laser pulses in a guard region of the FOV, each with an intensity below the threshold intensity. At step 21350 the detector detects a second set of laser reflections corresponding to the guard set of laser pulses and thereby generates second reflection data
At step 21360 the laser range finder performs the safety test again using the second reflection data and thereby generate a second result, and in response to the second result generates a second set of laser pulses in the adaptive-intensity region, each with an intensity below the threshold intensity. The second result can indicate the intrusion of an object (e.g. a person) into the adaptive-intensity region (e.g. the path of the high-intensity laser pulses) at some time in the near future. In several embodiments of method 21302, the laser range finder discontinues generating high-intensity laser pulses and instead exclusively generates laser pulses with intensities below the threshold intensity in the adaptive region, in response to the second result.
Exemplary safety tests can be: (a) a determination of any object is detected in the guard region, (b) a determination if any object in the guard region is moving towards the adaptive-intensity region, (c) a determination if any object in the guard region will intersect with a high-energy laser pulse or ingress into the adaptive-intensity region within a threshold period of time (e.g. a person will enter the adaptive-intensity region within the next 2 seconds), (d) a determination, based on reflection data from the set of guard laser pulses that an object exists in a guard region and within a threshold distance, or (e) a determination whether reflection data indicates an object in the guard region with an angular velocity (e.g. rate of change of direction in the FOV) above some threshold. Exemplary safety test results can be (a) satisfaction of a criterion (e.g. safety test result=TRUE), (b) dissatisfaction of a safety test (e.g. safety test result=FALSE), (c) an indication of a highest or lowest value (e.g. the closest proximity of an object to the adaptive intensity zone, such as result=10 meters) or (d) a velocity or angular velocity towards a keepout-zone for one or more objects.
Driving a vehicle often requires near-real time object tracking. In the process of driving a vehicle objects in the vicinity of the vehicle are often constantly changing location relative to the vehicle. For example, a vehicle driver who identifies a location estimate for a cyclist 21415a can instinctually associate an age with the location estimate indicative of the time elapsed since they estimated the location of the cyclist. When the age is low (i.e. the location estimate for the cyclist is very recent) the driver may perform a precise maneuver with the vehicle (e.g. crossing over an associated bicycle lane). Conversely, the driver may decide to be more cautious if the age associated with the cyclist location estimate becomes too large (e.g. the location estimate becomes greater than 5 seconds old).
Turning to
In the embodiment of
For each object in the set of objects the corresponding age and the corresponding location estimate can be used to generate a location probability distribution. The location probability distribution for an object (e.g. cyclist 21415a) can be a function or a database of probabilities such that for a candidate 2D or 3D location in the vicinity of the location estimate (e.g. location estimate 21410a) the location probability distribution can indicate a probability that the corresponding object (e.g. cyclist 21415a) occupies the candidate location at some time in the future. The location probability distribution can be based at least in part on a trajectory or direction of travel obtained for an object. For example, laser range finder 21420 can sense a greater velocity (e.g. rate of angular change in the FOV) for cyclist 21415a than pedestrian 21415b. Similarly, cyclist 21415a can be closer to the laser range finder and thereby subtend a larger range of angles per unit time. The laser range finder can calculate a perceived velocity for each object in the set of objects and use the perceived velocity to calculate the location probability distribution at some later time. For each object a threshold can be applied to the corresponding location probability distribution (e.g. a threshold that the probability of occupying a candidate location must be greater than 0.005). Laser range finer 21420 can determine for each object of the set of objects a corresponding object zone (e.g. portion of the surrounding vicinity) in which the location probability is greater than the threshold probability. Alternatively, an object zone corresponding to an object can be a set of 3D locations comprising a region within which the integrated probability of finding the object is greater than a threshold (e.g. the region in which there is a 95% probability of finding cyclist 21415a). For example, laser range finder 21420 can construct bounding box 21430a indicative of the object zone in which there is a 95% probability of finding cyclist 21415a at some time (e.g. at time=T1=2 seconds) after the location estimate 21410a. The bounding boxes 21430a and 21430b or similar object zones determined by a location probability threshold can have a 2D projection onto the FOV 21440, thereby generating corresponding object regions 21460a and 21460b within the FOV. Alternatively, laser range finder 21420 can calculate for each object an updated location estimate based on measurement data providing an initial location estimate, a trajectory and an age of the initial location estimate. In this way the updated location estimate for each object in the set of objects is a prediction of the present location of the object based on the initial location estimate and a measured trajectory.
Laser range finder 21420 can generate a set of laser pulses (e.g. pulse 21450) in a region 21475 of the FOV 21440. The intensity of each laser pulse in the set of laser pulses can be based at least in part on the corresponding location estimate (e.g. 21410a) and the corresponding age for at least one object in the set of objects in the vicinity. In an alternative embodiment each laser pulse can have an intensity based at least in part on a location probability distribution for an object. In yet another embodiment each laser pulses can have an intensity based at least in part on object zone (e.g. 21430a), an object region (e.g. 21460a or 21460b) or an updated location estimate for an object in the set of objects. In one embodiment of
In a similar embodiment laser range finder 21420 can identify that at time T1 the object regions 21460a and 21460b (e.g. the projections of object zones corresponding to objects onto the FOV) do not touch region 21475 in which the set of adaptive intensity laser pulses are generated and hence laser range finder 21420 can generate high-intensity laser pulses with directions in region 21475 of the FOV (e.g. laser pulse 21450).
In this way laser range finder 21420 uses the age of the location estimates to expand the zones of the vicinity (or regions of the FOV) where object are likely to exist. High-intensity laser pulses can have an initial intensity that is above an eye-safe threshold intensity and remain above the eye-safe intensity up to a threshold distance 21470. In the embodiment of
At step 21540 the method generates with the laser range finder a plurality of laser pulses, each comprising a laser pulse intensity from the set of laser intensities. At step 21550 the method detects with a detector in the laser range finder a plurality of laser reflections each corresponding to a laser pulse in the plurality of laser pulses
At step 21610 the method obtains location estimates for each object in a set of objects in the vicinity of a laser range finder. At step 21620 the method obtains for each object in the set of objects a corresponding age indicative of the time elapsed since the data indicating the location estimate of the corresponding object was gathered. At step 21630 the method generates for each object in the set of objects a corresponding location probability distribution, using the age and the location estimate for the object. At 21640 the method generates with a laser range finder a plurality of laser pulses, each with a laser pulse intensity based at least in part on the corresponding location probability distribution for an object from the set of objects. At step 21650 the method detects with a detector in the laser range finder a plurality of laser reflections, each resulting from at least one laser pulse in the plurality of laser pulses.
While the above description contains many specificities, these should not be construed as limitations on the scope of any embodiment, but as exemplifications of various embodiments thereof. Many other ramifications and variations are possible within the teachings of the various embodiments. Thus the scope should be determined by the appended claims and their legal equivalents, and not by the examples given.
Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like.
When a feature or element is herein referred to as being “on” another feature or element, it can be directly on the other feature or element or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being “directly on” another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being “connected”, “attached” or “coupled” to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being “directly connected”, “directly attached” or “directly coupled” to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed “adjacent” another feature may have portions that overlap or underlie the adjacent feature.
Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “/”.
Spatially relative terms, such as “under”, “below”, “lower”, “over”, “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures is inverted, elements described as “under” or “beneath” other elements or features would then be oriented “over” the other elements or features. Thus, the exemplary term “under” can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms “upwardly”, “downwardly”, “vertical”, “horizontal” and the like are used herein for the purpose of explanation only unless specifically indicated otherwise.
Although the terms “first” and “second” may be used herein to describe various features/elements (including steps), these features/elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature/element from another feature/element. Thus, a first feature/element discussed below could be termed a second feature/element, and similarly, a second feature/element discussed below could be termed a first feature/element without departing from the teachings of the present invention.
Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.
In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and/or steps may alternatively be exclusive, and may be expressed as “consisting of” or alternatively “consisting essentially of” the various components, steps, sub-components or sub-steps.
As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word “about” or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and/or position to indicate that the value and/or position described is within a reasonable expected range of values and/or positions. For example, a numeric value may have a value that is +/−0.1% of the stated value (or range of values), +/−1% of the stated value (or range of values), +/−2% of the stated value (or range of values), +/−5% of the stated value (or range of values), +/−10% of the stated value (or range of values), etc. Any numerical values given herein should also be understood to include about or approximately that value, unless the context indicates otherwise. For example, if the value “10” is disclosed, then “about 10” is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that “less than or equal to” the value, “greater than or equal to the value” and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value “X” is disclosed the “less than or equal to X” as well as “greater than or equal to X” (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.
Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.
The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.
This application is a continuation-in-part of U.S. patent application Ser. No. 16/459,494, filed Jul. 1, 2019, titled “MICROMIRROR ARRAY FOR FEEDBACK-BASED IMAGE RESOLUTION ENHANCEMENT,” now U.S. Patent Application Publication No. 2019/0324124, which is a continuation-in-part of International Application No. PCT/US2017/069173, filed Dec. 31, 2017, now International Publication No. WO 2018/126248, which claims the benefit of the following: U.S. Provisional Patent Application No. 62/441,492, filed Jan. 2, 2017, titled “DYNAMICALLY STEERED LASER RANGE FINDING FOR OBJECT LOCALIZATION,” and U.S. Provisional Patent Application No. 62/441,563, filed Jan. 3, 2017, titled “ELECTRONICALLY STEERED LIDAR WITH DIRECTION FEEDBACK,” and U.S. Provisional Patent Application No. 62/441,627, filed Jan. 3, 2017, titled “LASER RANGE FINDING WITH DYNAMICALLY CONFIGURED MICROMIRRORS,” all by the present inventor; the disclosures of which are fully incorporated by reference herein. This application is also a continuation-in-part of U.S. patent application Ser. No. 15/832,790, filed Dec. 6, 2017, titled “LIDAR WITH AN ADAPTIVE HIGH-INTENSITY ZONE,” now U.S. Patent Application Publication No. 2018/0106890, which is a continuation-in-part of International Application No. PCT/US2017/049231, filed Aug. 29, 2017, titled “LASER RANGE FINDER WITH SMART SAFETY-CONSCIOUS LASER INTENSITY,” now International Publication No. WO 2018/044958, which claims the benefit of U.S. Provisional Patent Application No. 62/380,951, filed on Aug. 29, 2016.
| Number | Date | Country | |
|---|---|---|---|
| 62441492 | Jan 2017 | US | |
| 62441563 | Jan 2017 | US | |
| 62441627 | Jan 2017 | US | |
| 62380951 | Aug 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16459494 | Jul 2019 | US |
| Child | 17114456 | US | |
| Parent | PCT/US2017/069173 | Dec 2017 | US |
| Child | 16459494 | US | |
| Parent | 15832790 | Dec 2017 | US |
| Child | PCT/US2017/069173 | US | |
| Parent | PCT/US2017/049231 | Aug 2017 | US |
| Child | 15832790 | US |
