This invention relates to efficient optical switches providing laser imaging of a field of view with scanning of the view provided without any mechanical movement through scanning through optical switching across an array of pixels that direct and receive light from a particular angle to an optical chip holding the array of pixels. The invention further relates to the components that provide this functionality and methods of implementing the no-movement imaging using coherent, frequency modulated continuous wave lasers and corresponding detection to obtain position and velocity information.
The ability to analyze and understand the 3D environment (3D Perception) is key to the success of robotic applications such as autonomous vehicles, UAVs, industrial robots, and the like. In mobile environments, 3D perception requires accurate and reliable object classification and tracking to understand current locations of objects as well as to predict their next possible move. See, Cho et al., “A Multi-Sensor Fusion System for Moving Object Detection and Tracking in Urban Driving Environments,” in 2014 IEEE International Conference on Robotics & Automation (ICRA), Hong Kong, China, May 31-Jun. 7, 2014. In applications such as autonomous driving car/UAVs, system may be required to identify and track many objects in real time. Thus, the ability to separate dynamic objects from the static ones can enable prioritization of processing tasks to identify and focus on regions of interest (ROI) leading to a faster response time. Light Detection and Ranging (LIDAR) is becoming a significant tool in the imaging context. See, published U.S. patent application 2016/0274589 to Templeton et al., “Wide-View LIDAR With Areas of Special Attention,” incorporated herein by reference.
One of the objectives of present disclosure is to introduce a method in which fast moving objects and their trajectories can be marked as region of interest (ROI) using a single LIDAR image frame. This ROI information then can be processed by machine vision algorithms for more accurate object classification and tracking. Unlike current methods of dynamic ROI identification, the methods described in the present application do not require use of large number of image frames to identify ROI of fast moving objects and their trajectory; depending on the relative speed of objects in the FOV, single image frame may be sufficient to identify ROIs corresponding to objects, their speed and trajectory. Multiframe approaches are described in Rogan, “Lidar Based Classification of Object Movement,” U.S. Pat. No. 9,110,163 B2, 18 Aug. 2015, Vallespi-Gonzales, “Object Detection for and Autonomous Vehicle,” U.S. Pat. No. 9,672,446B1, 6 Jun. 2017 and Rogan, “LIDAR-Based Classification of Object Movement,” Patent application US2016/0162742, all three of which are incorporated herein by reference.
Another objective of the present disclosure is to describe an integrated circuit that enables above mentioned ROI processing by taking advantage of coherent Lidar architecture implemented on photonic integrated circuit. Lidar IC described in this document enables 2D beam steering based on focal plane array vertical emitters with simple ON-OFF controls thus avoiding the complex analog controls of optical phase array based beam steering, including the issue of suppressing side lobes of the mean beam and having large far field beam size.
In a first aspect, the invention pertains to an optical chip comprising, a row of selectable emitting elements. The row of selectable emitting elements comprises a row feed optical waveguide, a plurality of selectable, electrically actuated solid state optical switches, a pixel optical waveguide associated with each optical switch configured to receive the switched optical signal, and a solid state first vertical coupler associated with the pixel waveguide. The solid state first vertical coupler is configured to direct the optical signal out of the plane of the optical chip. In some embodiments, the optical chip can comprise one or more additional plurality of rows of selectable emitting elements each comprising a row feed optical waveguide, plurality of selectable, electrically actuated-solid state optical switches associated with the row feed optical waveguide, a pixel optical waveguide associated with each optical switch, and a mechanically fixed, solid state vertical turning mirror associated with the target waveguide. For the additional plurality of rows of selectable emitting elements, the pixel optical waveguide can be configured to receive the switched optical signal, and the vertical tuning mirror can be configured to direct the optical signal out of the plane of the optical chip. In some embodiments, the optical chip can comprise a feed optical waveguide, a plurality of row switches to direct an optical signal along a row feed optical waveguide. In some embodiments, the optical chip can comprise multiple ports wherein each port is configured to provide input into a row.
In some embodiments, each pixel can comprise a balanced detector that is configured to receive light from the first vertical coupler. In some embodiments each pixel can comprise a solid state second vertical coupler and a balanced detector that is configured to receive light from the second vertical coupler. In some embodiments, each pixel can comprise an optical tap connected to the pixel optical waveguide and to a directional coupler. The directional coupler can be further connected to a receiver waveguide optically coupled to an optical splitter/coupler optically coupled to the first vertical coupler or optically coupler to the second vertical coupler. The balanced detector can comprise two optical detectors respectively optically connected to two output waveguides from the directional coupler.
In some embodiments, the chip can comprise a balanced detector and a directional coupler. The directional coupler can be configured to receive light from a second vertical coupler and from the row input waveguide. The balanced detector can comprise two photodetectors configured to receive output from respective arms of the directional coupler. The balanced detector can be within a receiver pixel separate from a selectable optical pixel.
In some embodiments, the selectable optical pixel further can comprise an optical tap connected to the pixel waveguide, and a monitoring photodetector configured to receive light from the optical tap. In some embodiments, the selectable optical switch can comprise a ring coupler with thermo-optical heaters. In some embodiments, the first vertical coupler can comprise a vertical coupler array. In some embodiments, the first vertical coupler can comprise a groove with a turning mirror. In some embodiments, the optical chip has silicon photonic optical structures formed with silicon on insulator format. In some embodiments, the optical chip has planar lightwave circuit structures comprising SiOxNy, 0≤x≤2, 0≤y≤4/3.
In a further aspect, the invention pertains to an optical imaging device comprising an optical chip and a lens. The position of the lens determines an angle of transmission of light from a selectable emitting element. In some embodiments, the lens covers all of the pixels, is approximately spaced a focal length away from the optical chip light emitting surface, and directs light from the selectable emitting elements at respective angles in a field of view. In some embodiments, the lens can comprise a microlenses associated with one selectable emitting element. The lens can further comprise additional microlenses each associated with a separate selectable emitting element.
In some embodiments, the optical imaging device can comprise an electrical circuit board electrically connected to the optical chip. The electrical circuit board can comprise electrical switches configured to selectively turn on the selectable optical switches. In some embodiments, a controller is connected to operate the electrical circuit board. The controller can comprise a processor and a power supply. In some embodiments, each pixel can comprise an optical tap connected to the pixel optical waveguide and to a direction coupler. The directional coupler can be connected to a receiver waveguide optically coupled to an optical splitter/coupler optically coupled to the first vertical coupler or optically coupler to the second vertical coupler. The balanced detector can comprise two optical detectors respectively optically connected to two output waveguides from the directional coupler. The balanced detector can be electrically connected to the electrical circuit board. In some embodiments, the optical imaging device can comprise an optical detector adjacent the optical chip. The optical detector can comprise a directional coupler optically connected to a vertical coupler, and a balanced detector. The balanced detector can comprise two photodetectors respectively coupled to an output branch of the directional coupler. The vertical coupler can be configured to receive reflected light from the optical chip and to an optical source from a local oscillator
In other aspects, the invention pertains to an optical array for transmitting a panorama of optical continuous wave transmissions comprising a two dimensional array of selectable optical pixels, one or more continuous wave lasers providing input into the two dimensional array, and a lens system. The lens system can comprise either a single lens with a size to cover the two dimensional array of selectable optical pixels or an array of lenses aligned with the selectable optical pixels. The lens or lenses can be configured to direct the optical transmission from the selectable optical pixels along an angle different from the angle of the other pixels such that collectively the array of pixels covers a selected solid angle of the field of view. In some embodiments, the two dimensional array is at least 3 pixels by three pixels, and wherein the two-dimensional array of optical pixels is on a single optical chip.
In some embodiments, the optical array can comprise at least one additional two-dimensional array of optical pixels arranged on a separate optical chip and configured with a lens system such that each optical chip covers a portion of the field of view. In some embodiments, each selectable optical pixel can comprise an optical switch with an electrical connection such that an electrical circuit selects the pixel through a change in the power state delivered by the electrical connection to the pixel. In some embodiments, the optical switch can comprise a ring resonator with a thermo-optic component or electro-optic component connected to the electrical connection. In some embodiments, the selectable optical pixel can comprise a first vertical coupler that is a V-groove reflector or a grating coupler. In some embodiments, the selectable optical pixel can comprise an optical tap connected to the pixel waveguide, and a monitoring photodetector configured to receive light from the optical tap. In some embodiments, the selectable optical pixel can comprise a balanced detector and a directional coupler that is configured to receive light either from the first vertical coupler or from a second vertical coupler, and to receive portion of light from the row input waveguide. The balanced detector can comprise two photodetectors configured to receive output from respective arms of the directional coupler
In a further aspect, the invention pertains to a rapid optical imager comprising a plurality of optical arrays, wherein the plurality of optical arrays are oriented to image the same field of view at staggered times to increase overall frame speed. In some embodiments, the plurality of optical arrays is from 4 to 16 optical arrays. The plurality of optical arrays can be optically connected to 1 to 16 lasers. The plurality of optical arrays can be electrically connected to a controller that selects pixels for transmission. In other aspects, the invention pertains to a high resolution optical imager comprising a plurality of optical arrays, wherein the plurality of optical arrays are oriented to image staggered overlapping portions of a selected field of view, and a controller electrically connected to the plurality of optical arrays, wherein the controller selects pixels for transmission and assembles a full image based on received images from the plurality of optical arrays.
In other aspects, the invention pertains to an optical chip comprising a light emitting pixel comprising an input waveguide, a pixel waveguide, an actuatable state optical switch, a first splitter optically connected to the pixel waveguide, a solid state vertical coupler, and a lens. The actuatable solid state optical switch can include an electrical tuning element providing for switching selected optical signal from the input waveguide into the pixel waveguide. The solid state vertical coupler can be configured to receive output from one branch of the splitter. The lens can be configured to direct light output form the vertical coupler at a particular angle relative to a plane of the optical chip.
In some embodiments, the optical chip comprises a first optical detector configured to receive output from another branch of the splitter, wherein the first splitter is a tap and wherein the first optical detector monitors the presence of an optical signal directed to the turning mirror. In some embodiments, the optical chip further comprises a second splitter configured between the first splitter and the turning mirror, a differential coupler configured to combine optical signals to obtain a beat signal from the first splitter and a received optical signal from the second splitter; and a balanced detector comprising a first photodetector and a second photodetector, wherein the first photodetector and the second photodetector receive optical signals from alternative branches of the differential coupler.
Moreover, the invention pertains to a method for real time image scanning over a field of view without mechanical motion, the method comprising scanning with coherent frequency modulated continuous wave laser light using a plurality of pixels in an array turned on at selected times to provide a measurement at one grid point in the image wherein the reflected light is sampled approximately independent of reflected light from other grid in the image points; and populating voxels of a virtual four dimensional image with information on position and radial velocity of objects in the image.
In some embodiments, the pixels can comprise optical switches that can be selectively turned on to project light along an angle specific for that switch. In some embodiments, detection of reflected light is performed using a balanced detector in the pixel, or using a balanced detector associated with a row of selectable pixels, or a detector adjacent the array of pixels. In some embodiments, a plurality of arrays of pixels are arranged to scan overlapping spaced apart portions of the field of view. In some embodiments, a plurality of arrays to scan of pixels are oriented to scan the same field of view to increase frame rate. In some embodiments, the scanning is performed with one laser wavelength. In some embodiments, the scanning is performed with a plurality of laser wavelengths. In some embodiments, Doppler shifts are used to determine relative velocity at each point in the image, wherein relative velocities and positions are used to group voxels associated with an object, and where the grouped voxels are used to determine the object velocity.
In other aspects, the invention pertains to a method for tracking image evolution in a field of view using a coherent optical transmitter/receiver, the method comprising: measuring the four dimensional (position plus radial velocity) along a field of view using a coherent continuous wave laser optical array; determining a portion of the field of view as a region of interest based on identification of a moving object; providing follow up measurements directed to the region of interest by addressing the optical array at pixels directed to the region of interest; and obtaining time evolution of the image based on the follow up measurements.
In some embodiments, the optical array can comprise pixels with selectable optical switches to turn on a pixel for emitting light along an angle in the field of view specific for the pixel. In some embodiments, detection of reflected light is performed using a balanced detector in the pixel, or using a balanced detector associated with a row of selectable pixels, or a detector adjacent the array of pixels. In some embodiments, a plurality of arrays of pixels are arranged to scan overlapping spaced apart portions of the field of view and/or are oriented to scan the same field of view to increase frame rate. In some embodiments, providing follow up measurements is performed by performing a scan using pixels with angular emissions for the pixels cover the regions of interest in the field of view. In some embodiments, the method can comprise performing additional scans of the full field of view interspersed with providing follow up measurements.
Optical arrays are configured with a plurality of addressable pixels on an optical chip in which the pixels are configured to emit lights outward from the surface with lenses arranged to direct the emitted light along a particular angle to the surface so that the array can cover a particular solid angle in the field of view. The systems use continuous wave laser light sources to perform coherent, frequency modulated continuous wave (FMCW) operation. The emitted light is generated by a coherent, continuous laser that outputs into waveguides along an optical chip with efficient electronically addressable optical switching to direct the laser light to a selected pixel. An optical chip can comprise a row of pixels with efficient switches, such as tunable ring resonators, and a pixel waveguide that directs the optical signal to a beam steering element that directed the optical signal from the surface of the optical chip, generally through a lens. Various appropriate configurations can be used for the detector. A pixel can comprise various splitters and combiners to tap off optical signals as reference for detection. The pixel can similarly be configured with optical detectors to function as a receiver with the split aperture (two beam turning elements) or a common aperture (single beam turning element), and two optical detectors in a pixel can operate as balanced detectors connected to a directional coupler with inputs connected to the beam splitters such that ne arm of the directional coupler has the received optical signal and the other arm of the directional coupler has the reference signal split from the optical input. In alternative embodiments, one receiver with balanced detectors can be used for a row of transmitting pixels, and in still further embodiments, a receiver can be separate form an optical chip performing the beam steering function. A plurality of arrays ot transmitters can provide wider ranges of the field of view and/or higher frame rates. Efficient and cost effective imaging systems can be designed that can provide effective applications in LIDAR systems.
Optical laser arrays power image generation and receiving that can provide for generation of extensive 4 dimensional data cloud with information on position and radial velocity of objects in the vision field. The ability to track the current position of objects and anticipate future positons is a significant objective of LIDAR that can enable improved autonomous vehicles. The advances described herein are based upon signal generation using one or a plurality of lasers with corresponding optics to provide projection and reception over a broad field of view without a movement-based scanning function. To effectively output the emissions from the laser array along appropriate output directions, a low loss optical switch array provides desired angle resolution. Effective switching functions are used to direct the optical signals along the selected row and column path. Individual pixels perform sending and receiving function to obtain data for the particular direction that is useful for the construction of the 4D image. A processor coordinates the image generation and processing of the image.
Traditional imaging can comprise a scanning function in which the light emitting and/or receiving elements are mechanically moved to scan a scene. To reduce the burden of moving larger elements, mirrors can be configured to steer the transmitted and received beams. Solid state beam steering without moving parts can greatly facilitate the scanning function by avoiding the mechanical motion to direct the beams. More generally, scanning technologies used in today's Lidar devices are either mechanical motion of optics or based on optical phased array techniques. Mechanical scanning is achieved either by rotation of the optical assembly or through mirror like reflector (i.e MEMS). Rotation based techniques are typically considered bulky, shorter life time and costly to manufacture. MEMS based scanner suffer from small FOV, lower frame rate and high sensitivity to mechanical shock and vibration. Optical Phase Array based beam scanning relies on large number of closely spaced optical elements and precise control of each element to direct the beam with low side lobes.
In a FMCW system, laser frequency is linearly chirped in frequency with a maximum chirp bandwidth B and laser output send to the target (Tx signal). Reflected light from the target is mixed with the copy of the Tx signal (local oscillator) in a balanced detector pair. This down converts the beat signal. Frequency of the beat signal represents the target distance and its radial velocity. Radial velocity and distance can be calculated when laser frequency is modulated with a triangular waveform, as described further below. This can be implemented in various ways with respect to scanning the field of view to construct the image. This is performed in the systems herein based on a solid-state beam steering array of pixels with appropriate optics to direct transmitted light to a grid over the field of view and with solid state optical switches performing the switching function. In the context of the discussion herein, stationary refers to the reference frame of the specific Lidar component, such as an optical chip such that it excludes components effectuated by movement, such as MEMS switches or mechanically scanned imaging components, which are not stationary with respect to the Lidar component, and the optical scanning device does not use internal motion for switching. Stationary switches are also sometimes referred to a ‘solid-state optical switches’. Both solid state and stationary, as used herein, refers to no internal motion in the optical scanning device as well as no scanning motion of the optical elements relative to the Lidar device. Thus, the optical switches and the pixel arrays are solid state, which reflects to non-moving parts aspect of the components and their function. Of course, the entire Lidar device can be part of a vehicle so that the entire Lidar system may be moving but herein this issue is not explicitly considered unless explicitly referenced.
Pixel based beam steering described herein allows for using less expensive lasers relative to techniques that rely on phase variance of the adjacent beams to provide a steering function through beam interference. Pixel based beam steering relies on the ability to create effective optical switches with low cross talk integrated along low loss waveguides on an optical chip. A received can be integrated into the chip to provide for a compact transmitter/receiver array. Control of the switching on the optical chip can be performed with an electronic chip, such as a CMOS integrated chip, that can be combined with the optical chip with appropriate aligned soldering. The readily scalable architecture can provide for high resolution and high frame rate.
Coherent Lidar (LIDAR based on FMCW) can provide depth and radial velocity information in a single measurement. Velocity information is obtained through the Doppler shift of the optical frequency of the return signal. In potential Coherent Lidar configurations, optical frequency of the laser can be modulated in a triangular form as shown in
A laser 100, such as a narrow line width laser, transmits an optical signal 101 which may be directly modulated by the laser, or the signal may be achieved through external modulator 103. The modulated signal passes through a lens 105 and reflects off target 107. Target 107 is located at a particular distance, or range, 109 from lens 105. If target is moving, it will also have a velocity 111 and trajectory 119. A time delayed optical reflected signal 113 returns through lens 105 where it is directed to a mixer 115, which can be a directional coupler that blends the received signal with a reference signal split from the optical input.
Frequency modulation of laser light can be archived through an external modulator or direct modulation of laser. Mixing the laser output (local oscillator) with the time delayed optical field reflected from the target generates a time varying intermediate frequency (IF) as shown in
Referring to
For machine vision applications, object classification involves image segmentation in which the voxels (volumetric pixels) in a 3D image frame or frames are identified as clusters of related voxels through methods described in the art. See, for example, Himmelsbach, et al., “LIDAR-based 3D Object Perception,” in Proceedings of 1st International national Workshop on Cognition for Technical Systems, 2008, Borcs, et al., “On board 3D Object Perception in Dynamic Urban Scenes,” in CogInfoCom 2013, 4th IEEE International Conference on Cognitive Infocommunications, Budapest, Hungary, Dec. 2-5, 2013, and Remebida, et al., “A Lidar and Vision-based Approach for Pedestrian and Vehicle Detection and Tracking,” in IEEE, Intelligent Transportation Systems Conference, ITSC 2007, all three of which are incorporated herein by reference. These methods use correlation of distance between voxels to create a cluster to segment the 3D image frame. A majority of these methods are very sensitive to model parameter selection and density of points in the image. In some methods, training is required, see U.S. Pat. No. 9,576,185 B1 to Delp, entitled “Classifying objects detected by 3D sensors for autonomous vehicle operation,” incorporated herein by reference. In most cases, a single image frame may not be sufficient to correctly identify a cluster of voxels that correspond to an object. In these cases, algorithms use multi-frame images to improve segmentation. Especially for object speed and trajectory, multi-frames image processing is required in current algorithms. For further discussion of image segmenting see Douillard, et al., “On the segmentation of 3D Lidar point,” in IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 2011.
Of all the sensors, Lidar plays an increasingly important role in 3D perception as their resolution and field of view exceed radar and ultrasonic sensors. In general, Lidar systems can be pulsed, phase coded or frequency modulated continuous-wave (FMCW) lasers. Pulsed Lidar operates by illuminating the scene by laser pulses (˜100 W peak power, ˜1 ns pulse width for 100-200 m range) and measuring the time of flight (TOF) of returned pulses. FWCM LIDAR on the other hand, use continuous wave laser output at low peak power and optically mix the return signal with the reference signal. Coherent mix of return signal and reference signal can provide simultaneously a large dynamic range and excellent ranging resolution.
Each image frame of LIDAR data includes a collection of points in three dimensions (3D point Cloud) which correspond to multiple TOF measurement within the sensors aperture (Field of view-FOV). These points can be organized into voxels which represents values on a regular grid in a 3-dimensional space. Voxels used in 3D imaging are analogous to pixels used in the context of 2D imaging devices. These frames can be processed to reconstruct 3D image as well as identify objects in the 3D image. 3D point cloud is a dataset composed of spatial measurement of positions in 3D space (x,y,z) corresponding to reflection points detected by LIDAR. Reflected light intensity from LIDAR is rarely used by classifiers as objects may be made of multiple materials with varying degree of reflectivity as well as environmental conditions/aging affecting the material reflectivity. Unlike pulsed laser based Lidar systems, Coherent Lidar (LIDAR based on FMCW-Frequency Modulated Continuous Wave) can provide depth and velocity information in a single measurement. Radial velocity information is obtained through the Doppler shift of the optical frequency of the return signal. In typical Coherent Lidar configuration, optical frequency of the laser modulated.
With the above mentioned measurements from the 4D LIDAR, an algorithm is presented that simplifies image segmentation. Based on the image segmentation and the 4D measurements, a lidar module can pre-process image frames and provide not only the X,Y,Z coordinates of Voxel, but also provide radial velocity information (Doppler shift frequency which is related to Voxel radial velocity) as well as segmented bin ID for Voxel s to indicate trajectories of objects in the field of view.
Motionless scanning can be performed with an optical array with light emitting pixels interfaced with a lens or array of microlenses that provide for aiming of output light form the pixels. Scanning the field of view based on the optical array is based on a low loss optical switch, which is described in detail herein based on micro-ring waveguide add/drop structure. One of the advantages of micro-ring add/drop configuration is its off-resonance pass through loss can be very low (i.e 0.001-0.01 dB) depending on the design and waveguide material. See, Bogaerts, et al., “Silicon microring resonators,” Laser Photonics Rev, vol. 6, no. 1, pp. 47-73, 2012, incorporated herein by reference.
A focal plane array consists of an input signal distribution bus section that distributes the input signal(s) to each row, optional modulator section and repeated pixel sections that act as a 1×N optical switch. Each Pixel is made of row signal bus, optical switch and vertical emitter. Light is emitted only when the optical switch in the pixel turned on. At a given time, only one pixel is turned ON in a given row while the other pixels in the same row are set to off. Multiple rows can be turned ON at the same time to enable column scanning instead of pixel-by-pixel scanning. In some embodiments, it can be desirable for the optical intensity to be almost all transferring into the pixel when the switch is on, while in other embodiments, it can be desirable for some residual intensity to continue along the row.
A micro-ring based switch can be turned on and off by adjusting its off-resonance frequency. Depending on the technology used, micro-ring resonance frequency can be changed by current injection, change in temperature or mechanical stress. Alternatively, input laser frequency can be tuned to micro-ring resonances to turn on a pixel or tune to off-resonance to turn-off a pixel. A signal input waveguide can operate as a row signal bus described in the figures described below for focal plane array, and a switch pass through port is connected to the next pixel in the same row. Total loss for the last pixel in the row is a function of number of pixels in the row and the waveguide length/loss. Thus, having extremely small pass through switch loss for each pixel reduces the total loss experienced by the last pixel in the row.
To enable light output from a pixel, drop port of the optical switch is connected to a vertical emitter. In some embodiments, a pixel can use a V-groove reflector to direct light out of plane. In this implementation, a V-groove is etched to waveguide and coated with partial or highly reflector. Partial reflector and photo detector may be used to monitor output optical signal level at the vertical emitter.
Even though a grating based vertical coupler can increase the complexity and introduce additional optical loss, their wavelength sensitivity can be used to fine tune the output angle. Both micro-ring switch operation and the focused grating emitter angle is a function of optical frequency. Thus, by changing optical frequency of the laser, output angle can be adjusted. This can result in finer angle tuning of the configuration of emitted light. In the case of focused grating vertical couplers, orientation of the grating structure can determine the direction of angular tuning at the output.
Coherent Lidar can only measure a single point in 3D space. In order to capture a 3D image of Lidar field of view (FOV), a transmitter beam is directed to different points of the grid within the FOV, which traditionally could be accomplished by scanning in 2D. Using the addressable pixel arrays described herein, each pixel can image a point in the FOV and high frame rates can be accomplished. The reflected light returning from the point projected outward is spread over an angular range such that collection of received reflected light may or may not be based on a receiver positioned adjacent the transmission location. Thus, received light can be collected at a convenient location, but generally based on the receiver location it is desirable to collect as much returned light as practical to improve signal to noise. A higher signal to noise for the receiver can improve the precision of the measurement. Embodiments of the switch based scanners described herein provide integrated receivers within the pixels of the transmitter, which provides a compact construction especially since the received light is referenced relative to a portion of the output light. In alternative embodiments, a receiver can be placed at the end of a row of transmitters to provide for ready access to a reference optical signal and provide for a somewhat larger receiver aperture. In further embodiments, a receiver can be placed adjacent to the transmitter array such that an even larger aperture can be used for the receiver, which simplifying the structure of the optical chip.
Using the beam steering arrays described herein, the field of view can be scanned by activating optical switches to turn on a particular pixel in the array that is structured to direct light along a particular direction. The turning on of the switch begins a measurement for that direction. If the light strikes an object it is reflected back along a cone of angles based on the relative amounts of specular and diffuse reflection as well as dispersion from propagation and scattering from particulates in the air, and other influences on the transmission. The distance to the object determines the time of flight for the returned light. For scanning over the array, one measurement is started by switching on a pixel, and integrating the detected signal over a measurement time such that the total for a pixel measurement is: ttotal=tswitch+tmeas. The frame rate is the time to scan over the entire field of view, which depends on the resolution, i.e., the number of array grid points. Roughly, a 250×250 grid of points over the field of view can be along the solid angle can be scanned in 16th the time of a 1000×1000 grid of points.
To increase the frame rate, multiple laser frequencies can be used, either through use of a tunable laser or using multiple fixed wavelength lasers tuned to different wavelengths, as long as the wavelength differences are larger than Doppler shifts due to object motion. The different laser frequencies can be simultaneously or overlappingly scanned if multiple detectors can be used to receive separately the different frequency transmissions. Various configurations of receivers described below allow for such scanning. In this way, the frame rate can be multiplied accordingly. Another way to multiply frame rate is to use a plurality of scanning arrays using the same or different wavelengths. If the arrays are sufficiently displaced from each other, the crosstalk between them can be sufficiently low that they can be used to scan the same or displaced portions of the field of view simultaneously or at least in overlapping measurement times. Examples of such embodiments are also shown below.
For scanning with one array with a particular wavelength, the time of flight for the light in getting to the object and reflecting back limits the measurement time. As noted in the previous paragraph, frame rates can be multiplied by using multiple wavelengths and/or using a plurality of scanning arrays. Also, the ability to dynamically control switching in the array can provide a power tool for the efficient improvement in resolution of particular regions of interest in the field of view. After performing a scan of the entire field of view, objects can be identified, moving and/or fixed, and some or all of these can be selected for a limited scan over the field of view. To scan over only a portion of the field of view, selected pixels can be identified, and such a limited scan can be performed over a correspondingly shorter period of time since the number of points scanned is correspondingly smaller than a full scan. Similarly, a full scan can be performed over a small resolution. For example, with a 1000×1000 array, a full scan can be performed over only a 250×250 set of pixels, which can be performed by skipping three of every 4 pixels in a row and three of every four rows in a column, such that the resolution is correspondingly smaller. Of course, the 1 in 4 example is only representative, and any lower resolution selections can be used as desired, such as 1 of 2, 1 or 3, . . . and the like. If the lower resolution scan identifies regions of interest, a higher resolution scan can be performed over the region of interest. The addressable array offers great flexibility for efficient yet effective scanning of the field of view.
In the present application, we present the following:
(1) A Lidar system generating 4 dimensional image (X,Y,Z for 3D location and V (radial) velocity) using a photonic integrated chip consists of a 2D beam scanner and a 2D coherent optical receiver having integral high speed switching function along with pixel selectivity.
(2) A Lidar image processing method that uses high frame rate 4D image with radial velocity information provided by single frame Lidar image to perform image segmentation for object classification and method of calculating object trajectory using a single Lidar image frame that can provide increased efficiency through identifying regions of interest that can be correlated with pixel; selection of the 2D scanner to allow specific increased monitoring of the regions of interest.
In dynamic environments, image pixels that belong to a moving object have similar radial velocity to each other regardless of the imaging perspective. Thus, use of radial velocity for clustering voxels in addition to their spatial proximity in a 3D point cloud image enables improved segmentation of the image and more accurately define object boundary. While these pictures are easy for a human brain to identify distinct objects, it is difficult for computer algorithms to make sense of the picture without a prior knowledge. In the case of
Transmitter portions of a Lidar optical circuit provides for output from each of an addressable array of pixels in which the pixels are structures to emit light along a particular direction within the field of view, in which the particular pixels generally direct light along different directions than other pixels. Collectively, the pixels can scan a grid along a solid angle of the field of view by sending and receiving optical signals from each pixel of the array that directs light along a particular grip point in the field of view.
The transmitter function relies on a focal plane array for 2D beam steering, as shown in
Referring to
Referring to
Corresponding receivers receive the reflected optical signals from the transmitters following interacting with the objects in the field of view. The receivers can be integrated with the transmitters into a single array, and efficient structures can be formed through integrating the receivers into the same pixels as the transmitters. Several embodiments of integral pixels with both transmitting function and receiving function are described below.
The transmitter/receiver arrays can be effectively formed from an optical circuit with integral optical switches that provide for addressable pixels. The optical switches can be controlled electrically, such as with resistive heaters that provide a thermo-optical effect, although other electrical induced index of refraction change can be implemented. Also, the receivers have electrical components that involve delivery of power and connection to processors. The optical circuit can be provided with metal contacts during formation that integrate the optical functionalities with appropriate electrical connections. The metal contacts can be furnished with solder balls to facilitate connection, such as to an electrical circuit board, a CMOS chip or other electrical chip structure.
An efficient electrical interface with the optical circuit can be established using a printed electrical circuit board, which can have aligned electrical contacts to interface with the electrical contact on the electrical circuit. The electrical connections with the optical chip electrodes can be made by wire bonding, but in some embodiments appropriate assembly can be performed using mated bonding pads on the electrical submount so that positioning of the optical chip with the electrical submount aligns the bonding pads on each that can then be connected, such as with reflow of solder. Since wire bonding balls would be placed at suitable locations, there can be no concern that they are conductive with no corresponding insulating structures between the elements. Other suitable processing approaches can be used. The electrical printed circuit board can be connected to appropriate processors and drivers.
As shown in
Referring to specific features of
Referring to
In embodiments, each row can have a row modulator between the input signal bus (waveguide) 739 and the row signal bus (waveguide) 737, although in
Referring to
Pixels generally are controlled through coordination of electrical signals and optical switches. Referring to
Referring to
In the embodiment shown in
Ring resonators can be formed using localized heating elements and trenches to isolate heat flow. These designs for the ring resonators can be more efficient and have faster response times. Efficient ring resonator designs as described herein are described further in published U.S. patent application 2020/0280173 to Gao et al. (hereinafter the '173 application), entitled “Method for Wavelength Control of Silicon Photonic External Cavity Tunable Laser,” incorporated herein by reference.
Referring to
In the embodiments shown in
Coherent Lidar is performed with FMCW (frequency modulated continuous wave) lasers. In general, tunable lasers can be used, or fixed wavelength lasers can be used, which can provide a cost savings. Modulated laser light should be provided to the pixels. The light can be provided by one or more lasers, and the configuration is influenced by the laser selection. With the use of low loss optical switches, correspondingly less laser power can be sufficient, although the optical circuit can include optical amplifiers as needed. Solid-state lasers can be effectively used to supply the laser power, although alternative lasers may be used. In some embodiments, the lasers can be integrated into the optical circuit. In other embodiments, a laser or array of lasers can be provides on a separate optical chip, and the laser optical chip can be optically connected with the optical chip forming the switched array functioning as transmitter/receiver.
Solid state tunable lasers are described, for example, in the '173 application cited above. A high power tunable silicon-photonics based laser is available from Applicant NeoPhotonics Corp. An array of separately controllable laser diodes is described in U.S. Pat. No. 9,660,421 to Vorobeichik et al., entitled “Dynamically-Distributable Multi-Output Pump for Fiber Optic Amplifier,” incorporated herein by reference. The laser power can be correlated with respect to the number of laser used, the number of pixels that may be powered simultaneously loss in the system and range for the imaging. In general, the laser powers are up to 100 mW (20 dBm), although more powerful lasers could be used.
Referring again to
Some lasers are suitable for direct modulation, in which the laser light output is modulated through control of the laser tuning. In other embodiments, external modulators are used. Suitable external modulators include, for example, electro-optic modulators. The electro-optic modulators can be formed through doping a section of the waveguide and attaching electrical contacts. The electro-optic modulator varies the phase, but through time dependent phase variation, the frequency is correspondingly modulated. So the electric signal driving phase variation is modulated according to the desired frequency modulation.
The selection of the number of lasers can be based on the size of the pixel array, the desired number of frames per minute, the desired range of the imager, laser properties and other design considerations. The number of lasers can be 1 or more than 1, in some embodiments no more than 100 lasers, but in generally the number of lasers is not generally constrained except by practical considerations of size and cost. The laser power can be from about 20 mW to about 5 W, in some embodiments from about 45 mW to about 2 W, and in other embodiments from about 75 mW to about 1 W. The lasers can be fixed wavelength solid state lasers, such as a laser diode—distributed feedback laser. While each array of pixels can be driven by a plurality of lasers, a single laser can effectively drive a plurality of arrays. Fixed wavelength lasers can be supplied at lower cost relative to tunable lasers. A person of ordinary skill in the art will recognize that additional ranges of laser power within the explicit ranges above are contemplated and are within the present disclosure.
The design of the laser interface with the pixel arrays can be guided by laser selection, number of pixels, and design of the switching functions. With one laser powering all transmittance, then the switching function accommodates all of the pixel selection and operation. A plurality of lasers can be used, which can either be fixed wavelength or adjustable wavelength and can be configured to transmit along the same waveguides or distinct waveguides from each other. If different wavelengths are directed along a common waveguide, these wavelengths can be multiplexed using an optical combiner, and wavelength selective switches along an array feed waveguide can be used for demultiplexing such that a particular wavelength can be directed down a row. In some embodiments, a single wavelength of light is directed down a feed waveguide, and a switch is activated to direct the light down a selected row.
In additional or alternative embodiments, lasers can be provided for each row. With this configuration, the rows do not need switches for row selection. The lasers can be connected abutting the optical chip with the laser coupled into the row waveguide, or any other reasonable connections for optical elements known in the art can be used, such as features for connecting optical fibers to optical chips.
To switch a pixel into an on state for transmitting and subsequent receiving, two switches can be placed in the on position, a row selector switch and a pixel selector switch. If the row has a separate input, there may not be a row selector switch. In general, it is most efficient to have the switch in a default off mode such that the switch is actuated to turn the switch on. Turning a switch on generally involves application of an electric current to engage some optical change, such as a change of index of refraction. A thermo-optical effect can be useful to effectuate this change of index of refraction, and ring resonators are described herein to operate as a low loss optical switch. A row selective switch can be a fixed wavelength selective switch or an actuatable switch analogous to a pixel selective switch. Alternatively, a row can have a dedicated input so that only a pixel switch is turned on to direct light into the pixel for transmission in the selected direction.
Suitable vertical emitter elements can be a mirrored V-groove. Referring to
Specifically, four exemplary embodiments of V-groove reflector 1009 are shown in
In a second embodiment 1009.2, V-groove 1011 has a non-reflective face 1015 allowing optical signal 1001 to pass through, and a reflective face 1017 that directs optical signal 1001 to exit the pixel away from the substrate. While reflective face 1017 can be metalized, it can be less desirable than alternative structures since non-reflective face 1015 should be free of metal to be highly transmissive. In embodiments, V-groove 1011 may be filed with an appropriately shaped deposit of reflective polymers to form reflective face 1017. Accordingly, various pixel orientations within an array are achievable with changes to the orientation of V-groove reflector 1009.
In the third embodiment shown 1009.3, substrate 1013 of pixel 1000 is patterned to create integrated lens 1019.1 aligned with V-groove 1011 such that optical signal 1001 exits lens 1019.1 in a collimated beam 1021 having no angular offset. In the fourth embodiment shown, substrate 1013 of pixel 1000 has integrated lens 1019.2 which is offset from V-groove 1011 by a distance Δd 1023. The offset Δd 1023 causes collimated beam 1025 to exit from pixel 1000 with angular offset θ 1027. Integrated lens 1019.2 may or may not be a spherical lens. For a spherical lens, the relationship between Δd 1023 and angular offset θ 1027 may be characterized in θ=atan(Δd/f), where f is the focal length of lens 1019.2 and other shaped lenses can be used to achieve desired angular propagation as determined by geometric optics.
An embodiment of a polymer based turning mirror suitable for the embodiment of the V-groove vertical deflector is shown in
In alternative embodiments, surface grating couplers can be used to perform vertical turning of the optical path. A representative surface grating coupler is shown in
Standard silicon photonics surface gratings are on the rough order of magnitude of tens of microns or less, possibly into single digits of microns. To support further shrinking the pixel size, a more compact method of launching the light vertically is desirable. The grating coupler shown in
In an alternative embodiment, as shown in
A particularly compact structure combines the receiving function with the transmitting function within a pixel. This can provide a structure in which the transmitted signal can be split for use as a reference signal for evaluating the received signal in a short span of waveguide. Either the vertical emitter element for transmission is used for receiving or a parallel structure is used for receiving, which can be located adjacent the transmitter element. In either case, the elements can use the same lens.
Thus, a pixel within the array of pixels can comprise an optical switch to turn on the pixel with respect to receiving input laser light. The input light can be split with a portion of the light directed to a receiver to provide a reference for the received signal. Optionally, a tap can remove a portion of remaining signal to direct to a receiver component, such as a photodiode for monitoring. The monitoring receiver can confirm activation of a pixel. The remaining light signal can be directed to a vertical coupler. As noted above, the same vertical coupler can be used as a receiver, or a separate adjacent vertical coupler can be used for receiving a signal. The received signal then is transmitted back to an optical combiner/splitter (for the single aperture configuration) that directed at least a portion of the received optical signal toward the balanced detectors or directly toward the balanced detectors. The received optical signal is directed to a directional coupler that is also connected on its other input to the input reference signal. The two outputs of the directional coupler are directed, respectively, to one of the two photodetectors of the balanced receiver. The pixel can be coupled to appropriate electrical connections that control the optical switch to turn the pixel on, the optical detectors and optionally the optical monitor.
The receiving function is shown in
A balanced receiver incorporated into the pixel allows each pixel to act as a coherent receiver as well as a directional transmitter, although the receiver elements can be separate from the steering transmitter array. The coherent receiver receives optical signals from the convolution of a reference signal associated with the local oscillator (i.e., the laser source) and the return signal. In embodiments, as shown in
As shown in
Referring to
Pixel 1400 comprises electrical contacts for connection with an overlaid optical circuit, such as provided by a circuit board, and
The pixel dimensions generally dictate the overall chip size, which will impact the fabrication yield as well as the size and optical performance which may influence an interface with free space coupling optics. Larger pixels can involve longer propagation distances which reduce output power, range and sensitivity. Reductions in area and can be realized through careful component optimization.
In a FMCW system, laser frequency can be linearly chirped in frequency with a maximum chirp bandwidth B and laser output sent to the target (Tx signal). Reflected light from the target is mixed with the copy of the Tx signal (local oscillator) in a balanced detector pair. This down converts the beat signal. Frequency of the beat signal represents the target distance and its radial velocity. Radial velocity and distance can be calculated when laser frequency is modulated with a triangular waveform. The modulation of the laser frequency can be according to a triangular wave form, as shown in
The up beat frequency and the down beat frequency give the distance and radial velocity Mixing the laser output (local oscillator) with the time delayed optical field reflected from the target generates a time varying intermediate frequency (IF) as shown in
Range=((fdiff_down+fdiff_up)/2)·(T·c)/(4·B). (1)
The two intermediate frequencies, fdiff_down and fdiff_up) are obtained from the Fourier transform of the signals received by the two receivers and selecting the center frequencies corresponding to the peak of the power spectrum in the Fourier transform. For the case of a moving target, a Doppler frequency shift will be superimposed to IF (shown as change in frequency over the waveform ramp-up and decrease during ramp-down, see
Doppler Velocity (VD)=((fdiff_down−fdiff_up)/2)·λ/2), (2)
f
IF=(f+IF+f−IF)/2=((fdiff_down+fdiff_up)/2). (3)
where λ is the laser wavelength. The object velocity (V) is evaluated as VD/Cos(ψ2), where ψ2 is the angle between the laser beam direction for an edge of the object and the direction of motion, which is described further below. The beat frequencies can be extracted from Fourier transforms of the sum of the current as a function of time from the balanced detectors using known techniques from coherent detection.
The range and radial velocity information can then be used to populate the voxels. The distance is determined within a particular resolution. Resolution (ΔR): Describes the minimum distance between two resolvable semi-transparent surfaces.—Semi-transparent surfaces closer than minimum distance will show up as a single surface. Resolution is inversely proportional to tuning bandwidth ΔR=0.89 c/B. The distance determination is also evaluated within a particular precision or numerical error. Precision (σR): Describes the measurement accuracy and depends on received signal SNR and chirp bandwidth. In most systems, Precision (σR)>>Resolution (ΔR) and is determined by σR=c/(4πB)(3/SNR)1/2, where SNR is the signal to noise ratio.
In FMCW system design, laser chip bandwidth can be selected to meet the system precision requirements. Generally, a SNR of at least about 13 dB is used, which translates to σR=0.93 cm precision for B=1 GHz. For higher precision, laser chirp bandwidth can be increased. This precision value represents the worst case value at the lowest value of SNR, for closer targets or targets with higher reflectivity, the receive signal SNR increases, and thus the precision improves. For example, for the same chirp bandwidth of 1 GHz, if SNR increases from 13 dB to 30 dB, precision increases from σR=0.93 cm to σR=0.13 cm. Note if higher precision is desired then, chirp bandwidth can be increased.
In dynamic environments, image pixels that belong to a moving object a have similar Doppler (radial) velocity regardless of the imaging perspective, although the value of the Doppler (radial) velocity is a function of the angle, as explained below. Thus, use of Doppler (radial) velocity for clustering voxels addition to their spatial proximity in a 3D point cloud image enables improved segmentation of the image and more accurately define object boundary. Based on this principle, with populated voxels, objects can be identified. In particular, neighboring points in the angular distribution at approximately the same range and traveling at the same speed can be grouped as part of the same object. Correspondingly, identification of the object provides for backing out the trajectory from the Doppler velocities. The process can be organized into the following algorithm.
1. Identify the number of Velocity Bins (Vi) in Image frame:
a. Use Vi+/−ΔV for each cluster, where ΔV is variation of radial velocity in each cluster
2. For each (radial) Velocity Bin Vi
a. Using spatial clustering techniques such as GNM (Gaussian Noise Model), K-NN (K-Nearest Neighbor) or CNN (Convolutional Neural Networks), define object boundaries. This operation is used to segment objects with similar doppler (radial) velocity in adjacent spatial positions.
Above algorithm can be used to quick identification of dynamic objects in a single frame without use of information from other image frames.
Estimation of Object Trajectory and Speed from a Single Lidar Frame with 4D Data
In coherent Lidar, Doppler shift is related to radial velocity of the point being measured and trajectory This is schematically laid out in
The angles γ, θ1, θ2, ψ1, ψ2 are shown in Fig. A, and ψ1=γ+θ1, and ψ2=γ+θ2. Also, Vd1=V0+cos(ψ1) and Vd2=V0+cos(ψ2). Unknowns V0 and γ can be evaluated from known Vd1, Vd2, θ1, and θ2. This can be generalized to three dimensions using a third point of the three dimensional position image and the radial velocity of the third point.
Referring to
Referring to
The vertical array switching devices described herein generally rely on scanning each pixel by turning on or off a low loss switch within the pixel. In the simplest configuration of an N×M pixel array, frame rate scales with the total number of pixels in the array. Sequential scanning of each pixel in a large array reduces the frame rate.
An alternative embodiment, as shown in
By way of example, in order to support a 600×400 pixel array, 16 vertical switch arrays can be used. In a configuration where the laser output is then split 16 ways to power each of the vertical switch arrays, the combined 600×400 pixel resolution can scan at a rate of 20 frames/sec. Following these examples, it is clear how to adjust the array size to achieve the frame rate of an image sensor.
With a single vertical coupling array, it is only possible to turn on a single transmitting pixel in a row at a time for each laser frequency to allow for measurement of the reflected signal. If a vertical coupling array is connected to polychromatic light, either multiplexed or not, the different laser frequencies can be scanned separately for transmission and reception. Alternatively, if different laser light sources are configured to send optical signals down different rows of sets of rows, these can be separately scanned if there is sufficiently low crosstalk between the signals. The scanning of the pixels may not proceed linearly along a grid, and based on switching times (on and off), less noise, lower cross talk and shorter scan times may, in some embodiments, occur if sequential on pixels may be spatially separated. On the other hand, for focused scanning a region of interest, sufficient scanning efficiencies are gained from the limited focused scans that sequential scans in adjacent pixels can be very efficient even if per scan rates may be slowed somewhat.
The Lidar systems described herein provide considerable flexibility and efficiencies that allow for adaptation or selection of alternative operation cycles depending on the observed circumstances. Parameters that can influence selection of scanning protocols can include: distance of objects, speed of object motion, signal to noise of reflected signal, and the like. While signal to noise ration depends on object distance and transmitted laser power, as described above, it can also depend on reflectivity of the object and weather conditions, for example, rain or snow can scatter significant amounts of outgoing and reflected light. The ability to have a wide range of adjustability with virtual instantaneous programing ability is a great advantage.
The embodiments above are intended to be illustrative and not limiting. Additional embodiments are within the claims. In addition, although the present invention has been described with reference to particular embodiments, those skilled in the art will recognize that changes can be made in form and detail without departing from the spirit and scope of the invention. Any incorporation by reference of documents above is limited such that no subject matter is incorporated that is contrary to the explicit disclosure herein. To the extent that specific structures, compositions and/or processes are described herein with components, elements, ingredients or other partitions, it is to be understand that the disclosure herein covers the specific embodiments, embodiments comprising the specific components, elements, ingredients, other partitions or combinations thereof as well as embodiments consisting essentially of such specific components, ingredients or other partitions or combinations thereof that can include additional features that do not change the fundamental nature of the subject matter, as suggested in the discussion, unless otherwise specifically indicated. The use of the term “about” herein refers to imprecision due to the measurement for the particular parameter as would be understood by a person f ordinary skill in the art, unless explicitly indicated otherwise.
This application claims priority to copending U.S. provisional patent application 63/159,252 filed Mar. 10, 2021 to Canoglu et al., entitled “Method of Improved Object Classification Based on 4D Point Cloud Data from Lidar and Photonic Integrated Circuit Implementation for Generating 4D Point Cloud Data,” incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
63159252 | Mar 2021 | US |