Patent Application
20070019181
The field of the invention relates range finders, collision avoidance systems, automated object detection systems, optical proximity detectors, and machine vision.
As technology has advanced over the years, more and more automated means have been developed to do tasks which were originally accomplished by human beings. Indeed, automation and machinery have made possible the accomplishment of many things which human beings could not do without automation and machinery. At one level, tasks have been automated by making special-purpose machines and/or special-purpose software which do particular tasks. At another level, machines and software have been designed which automate the running of other machines and software.
One of the frontiers of modern automation is the automation of tasks which have traditionally relied on human visual perception. In an agricultural example, many tasks are currently accomplished by people running fairly complex mobile machines, where the job of the person has often been reduced to simply navigating the machine from place to place and controlling the machine with simple controls to perform different tasks.
Technology is currently being developed to automate many agricultural tasks to an even higher level, by providing autonomous guidance mechanisms for automated machines, such that human beings will not need to be present for a large fraction of the time the machine is operating, including times when the autonomous machine is moving from one place to another.
One of the major challenges facing the designers of autonomous agricultural machinery is the design of systems which allow autonomous machinery to intelligently navigate from place to place in real-world environments. When a human being navigates a machine from place to place, the human being utilizes the ability to recognize patterns and objects, such as roadways, intersections, and obstacles along a path, and respond appropriately.
If the physical environment through which an autonomous vehicle needs to navigate is well-known and specified, an effective guidance system can be far more economically designed. Unfortunately, unexpected changes to the environment occur frequently in the real world. In an agricultural environment, unexpected obstacles that might be encountered include parked cars, tools and machinery left in the wrong place, barrels, and fallen branches.
The agricultural industry needs inexpensive, highly physically robust systems for detecting obstacles in the path of autonomous vehicles. It is an object of the present invention to provide a highly mechanically robust, inexpensive obstacle detection system which is suited for use on autonomous agricultural machinery.
In a home automation example, it may be desirable for a domestic robot to be able to navigate within a home, avoiding obstacles such as furniture, walls, plumbing fixtures, appliances, and people, and negotiating stairs.
In another home automation example, it may be desirable for a domestic robot to be able to perform a security function, such as monitoring a room to detect intruders, or keeping pets off of counter tops or furniture.
In one embodiment, the present invention uses a rugged, inexpensive laser diode and a beam splitter to project a structured light pattern in the form of an array of co-originating beams of light forward from the front of in an autonomous vehicle at a downward angle, such that the beams intersect the ground a known distance in front of the vehicle. A video camera which is not co-planer with the projected beam array observes the intersection of the beam array with objects in the environment. The height of the beam spot images in the video image varies with distance of the intersected object from the autonomous vehicle. The forward-projected beams traverse the obstacle from bottom to top as the vehicle moves forward. Triangulation is used to measure both the height and distance from the vehicle at which each forward-projected beam intersects either the ground or an obstacle, so that the vehicle can either maneuver around obstructions or stop before colliding with them.
The projected beams of light are modulated at a known frequency, and the observed video images are synchronously demodulated to provide an image insensitive to ambient lighting conditions.
In a preferred embodiment, two (approximately spatially coincident) video cameras with partially overlapping fields of view are used to get a wider forward-looking field of view and/or better angular resolution while still using standard commercial modules. The system has no moving parts and can operate reliably under significant shock and vibration conditions.
In another embodiment, the present invention acts as a collision avoidance alarm and/or automated emergency braking system on railed vehicles such as trains and subway cars.
In another embodiment, the present invention provides navigation aid to a self-navigating domestic robot. In this embodiment, the optical and electronic apparatus affixed to an autonomous domestic robot. In this and other embodiments used on autonomous vehicles, the present invention may incorporate dead-reckoning hardware and mapping software. In such an embodiment, the present invention allows an autonomous vehicle to inexpensively map out its environment high degree of accuracy. Dead reckoning means contemplated to be incorporated into the present invention includes ground-contact forms of dead reckoning such as wheels, and non-contact forms of dead reckoning such as GPS and optical odometry, as described in co-pending patent application Ser. No. 10/786,245, filed Feb. 24, 2004 by Sinclair et. al., which is hereby incorporated by reference.
In a preferred embodiment, subsequent to the initial mapping of the environment, the amount of processing power needed to detect changes to that environment and re-map detected changes is significantly less than the amount of processing power needed to form the original map. The majority of objects mapped (such as walls, furniture, plumbing fixtures, and appliances will rarely move and thus rarely need to be re-mapped, whereas the position of doors, kitchen and dining room chairs, etc. may move frequently. This efficient utilization of computational resources inherent in partial dynamic re-mapping can allow for lower power consumption and cheaper implementation of domestic robots. In addition, utilization of dead-reckoning systems in conjunction with object detection can result in far more computationally efficient navigation once an area or operation has been initially mapped.
In another embodiment, the present invention uses multiple structured light patterns projected from a fixed position to measure changes in object positions within a pre-determined “keep-out” volume of space over time. In this embodiment, a training mode is provided in which the present invention learns the perimeter of the keep-out volume as an object is three-dimensionally moved around the imaginary surface which defines the keep-out volume. One specifically contemplated application for such an embodiment is use in security systems. Another application specifically contemplated is domestic use to train pets to stay off or away from cherished objects and furniture.
It is an object of the present invention to provide a mechanically robust, inexpensive method and apparatus for obstacle detection for use on autonomous vehicles. It is a further object of the present invention to provide an inexpensive optical security device capable of detecting unwanted movement or presence of objects within a monitored volume of space. It is a further object of the present invention to provide an inexpensive, mechanically robust, reliable vehicle collision avoidance system. It is a further object of the present invention to facilitate inexpensive self-navigating domestic robots.
In
Video camera 2109 views any object intersecting light beam 2102 at distance D1 along line of site 2103. Video camera 2109 views any object intersecting light beam 2102 at distance D2 along line of site 2104. Video camera 2109 views any object intersecting light beam 2102 at distance D3 along line of site 2105. Video camera 2109 views any object intersecting light beam 2102 at distance D4 along line of site 2106. Video camera 2109 views the ground intersecting light beam 2102 at distance D5 along line of site 2107.
As autonomous vehicle 2100 moves forward an obstacle in its path would first be illuminated by light beam 2102 at distance D6 in front of the vehicle. As the vehicle moves closer to the object the illumination spot which light beam 2102 projects on the obstacle traverses the obstacle vertically from bottom to top. While
In a preferred embodiment of the present invention which utilizes multiple sets of light beams intersecting the ground at progressively further distances from the autonomous vehicle (as illustrated in
As the autonomous vehicle moves forward, the observed intersection of non-centrally projected beams not only traverses objects vertically as the vehicle moves forward, the image also traverses intersected objects horizontally. In one preferred embodiment, non-centrally-directed projected split beams are tightly focused to improve signal-to-noise ratio, and non-centrally located thin objects are detected by observing the image often enough so that the image of a spot traversing any object horizontally will always be observed. In such an embodiment, centrally located beams are given some overlap to avoid missing thin vertically-oriented centrally located objects which could otherwise be missed (because there is no apparent “sideways” motion of centrally projected beams across the field of view of the video camera as the beam traverses an obstacle due to forward motion of the vehicle.
In order to reduce sensitivity to ambient lighting conditions, in a preferred embodiment of the present invention, the projected light beams are modulated and the observed video signal is synchronously demodulated. Since the video image is inherently sampled at the frame rate of the video, it is convenient to phase-lock the modulation of the projected light beams with the video sampling rate. For example, if the video sampling rate is 60 frames per second, a preferred embodiment of the present invention utilizes light beams that are square-wave-modulated at 30 Hz, such that the square-wave transitions in the beam intensity occur simultaneously with the time boundaries between successive video captures. In such an embodiment, the beam pattern could be said to be present in every even numbered video capture, and absent in every odd numbered video capture. By taking the difference between successive video captures (or multiplying the brightness of each pixel successively by +1 and −1) and averaging the result, the intersections of the projected beams with objects in the environment stand out in high contrast to the remainder of the image.
It is important to keep dirt from getting on the optics of the system, and for sytems operating in an agricultural environment (which is replete with sources of dirt, mist, chemicals, etc.), to prevent the optics from accumulating dirt or liquid or chemical coatings which could impair performance, in a preferred embodiment of the present invention, the beam projecting and video optics are recessed in open-window chambers which are connected to a positive-pressure air supply. The optics thus “looks out” through an opening which always has air flowing out through it, at a rate sufficient to prevent most dirt particles, moisture, chemicals, etc. from coming in contact with the optics. In an alternate preferred embodiment, a rotating window may be used in conjunction with affixed sprayer and wiper to keep dirt out of continuously used optics. In an alternate preferred embodiment, an automatic intermittent sprayer and an automatic intermittent wiper may be used to keep dirt out of the optics where the optics are intermittently used.
It is contemplated that alternate embodiments of the present invention could use beam scanning technology (such as the spinning mirror technology used in laser printers and check-out counter bar-code readers). In embodiments of the present invention utilizing scanning optics in place of a beam splitter, the advantage of continuous optical striping in captured images (which avoids missing “thin” objects in single images) can be traded off against the advantages of reflected optical power inherent in projecting spots instead of stripes.
In determining the position of objects, the fundamental principal on which the present invention relies is triangulation. Some methods of using structured light in conjunction with one or more electronic imagers to perform triangulation are described above. Other methods contemplated include projecting multiple simultaneous structured light patterns of different colors, multiple spatially interspersed and spatially distinguishable structured light patterns, and multiple temporally distinguishable structured light patterns. For instance the angle of a planar structured light pattern over time, between capturing a plurality of images. This embodiment may be particularly useful in applications where the structured light projector and imager remain fixed and it is desired to monitor object movement within a volume of space over time, such as security applications or pet-training applications. The triangulation of the present invention may be accomplished with a single imager and a single projecting aperture, multiple imagers and a single projecting aperture, multiple projecting apertures and a single imager, or multiple projecting apertures and multiple imagers.
Some varied embodiments of the present invention are depicted in
It may also be seen that the vertical position of light spots found within image boundaries 2214 is indicative of the radial distance of those objects from imager 2206. Thus, light spots found at height 2212 within image frame 2214 would come from intersections of light beams wit objects at distance D1, while light spots found at height 2213 within image frame 2214 would come from intersections of light beams with objects at distance D2.
In some preferred embodiments, it may be desirable to gain enhanced distance resolution around some distance in the field of view. With the embodiment depicted in
It may be desirable in some applications of the present invention to have the ability to detect objects within a three-dimensional volume, rather than just detecting the intersection of objects with a two-dimensional structured light pattern. This may be accomplished through detecting the intersection of objects with multiple planar structured light patterns, where the planes of the multiple patterns are oriented at different angles, as shown in
In an alternate embodiment, left-to-right angular resolution is traded off against vertical resolution. In such an embodiment, the beams of the multiple planar structured light patterns are horizontally interlaced as shown in
In an alternate embodiment where objects in the field of view can be assumed to remain relatively still over some short period of time, multiple planar structured light patterns of differing angles may be projected sequentially in time.
Although the preferred embodiments depicted in
A top view of a multiple-co-planar-imager single-coplanar-structured-light-pattern embodiment of the present invention is depicted in
In a preferred embodiment of the present invention, processing of multiple images is used in place of processing of a single image, to improve signal-to-noise ratio through averaging techniques, and techniques or removing from a set of images to be averaged any image with significantly outlying data. In a domestic application, statistically outlying images might be acquired when a flying insect flew near the optical aperture from which the structured light pattern originates. In an agricultural application, a statistically outlying image might be acquired when debris blows in front of the structures light source aperture, or when dirt or liquid momentarily corrupts the surface of the optical aperture before being automatically removed.
In a preferred embodiment of the present invention, the re-locating of objects from various vantage points at various distances is used in the mapping process to build an object map with more consistent spatial accuracy than would be possible in mapping from a single vantage point. Since the error in triangulation is angular, the absolute distance resolution gets linearly worse with radial distance from the imager. Imaging from multiple vantage points at a plurality of distances overcomes this limitation.
In a preferred embodiment of the present invention, object mapping is done utilizing varying spatial resolution, such that objects with large approximately planar surfaces are represented with few data points and objects with more rapidly spatially varying features are represented with more data points. In a preferred embodiment, the re-mapping of the position of known objects is done in such a way that the most rapidly spatially varying portions of objects that have moved take more computation time to re-map, while the less rapidly spatially varying portions of objects take less time to re-map. This mapping architecture inherently represents the edges of objects with greatest accuracy, as would be desired for navigation purposes.
The storage means used to store map data and image data in the present invention may be any type of computer memory such as magnetic disk, RAM, Flash EEROM, optical disk, magnetic tape, and any other type of memory as may come into use over time for computational purposes. The means for digitally processing acquired images in the present invention can be any type of microprocessor, computer, digital signal processor, array processor, custom application-specific integrated circuit (ASIC), state machine, or the like. The electronic imagers used in the present invention may be any type of electronic camera, video camera, liner or two-dimensional imaging array such as a CCD array, COMS array, or the like.
The foregoing discussion should be understood as illustrative and should not be considered to be limiting in any sense. While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the claims.
This application claims priority to provisional patent application No. 60/463,525, filed Apr. 17, 2003.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US04/12295 | 4/19/2004 | WO | 10/17/2005 |
| Number | Date | Country | |
|---|---|---|---|
| 60463525 | Apr 2003 | US |
