Patent Application
20120287239
This invention relates to robot vision, and more particularly to using thermal imaging to provide three dimensional robot vision.
With the rising use of robots for various applications, the need exists for improved robot vision technology. Robot vision enables robots to be operated under a wide range of conditions and to have information that maximizes situational awareness. Robot vision is important for accurate and fast navigation through complex terrains, for better interpretation of the robot's surroundings, for rapid recognition of objects of interest, and for improved ability to interact with objects with manipulators or by other means.
For robot vision, a combination of cameras or other image detection equipment and vision algorithms are used. Robot vision systems can range from a simple guidance application to a more complex application that uses data from multiple sensors.
A common implementation of robot vision uses a single visible light video camera. Typically, a monoscopic image is displayed on a monitor viewed by the robot operator. There are a number of limitations to this approach. Because the video is monoscopic, it does not provide depth or shape information other than what can be gleaned for the relative position of objects in the image. Also, the quality of visible and near-infrared video varies significantly depending on illumination levels and direction. High sensitivity image intensifiers can improve video quality under low light conditions, but become saturated if an illumination source enters the field of view and may not even work in full sunlight. Onboard light sources can compensate for low lighting, but require more power, have limited range, and would reveal the robot's location in covert operations.
A more complete understanding of the present embodiments and advantages thereof may be acquired by referring to the following description taken in conjunction with the accompanying drawings, in which like reference numbers indicate like features, and wherein:
The following description is directed to a three-dimensional stereo robot vision system, which provides a robot operator with enhanced situational and positional awareness by the use of long-wave infrared (LWIR) images. The system works well under almost any lighting condition, day or night, and provides the operator with information about the robot's surroundings that could not be determined with traditional imaging systems. The system has at least one camera mounted on the robot, an optical system, a means to transmit image data to the robot operator, and a means for the operator to view three-dimensional stereo images.
Long wave infrared (LWIR) is a useful and versatile wavelength band (8-12 um) that requires no illumination, provides information about objects and terrain not available in visible and near-infrared images, and is not affected by typical light sources. However, one difficulty with using LWIR for night vision is image interpretation. Scenes and objects can appear quite dissimilar between LWIR and visible images, with radiative emission being the dominant source of the apparent differences rather than reflection or absorption. Common objects can be challenging to recognize or have an unexpectedly low contrast. An additional difficulty is that LWIR cameras tend to be of low resolution.
One thing that does not change from the visible to LWIR is the dimensional shape of objects. Adding dimensional information to the image greatly improves the ability to interpret them.
One embodiment of the invention uses two cameras on-board the robot. Two identical LWIR cameras with equivalent optical systems are placed side by side at an appropriate horizontal separation to achieve the desired stereo effect. However, as explained below, in other embodiments, the three-dimensional stereo robot vision may be achieved using only a LWIR single camera and a specially designed optical system. In all embodiments, the camera may provide still images or a stream of images, such as a video stream.
As explained below, system 100 provides two images (or two sets of video streams) whose fields of view are spatially separate in a horizontal plane such as are human eyes. More specifically, camera 11 and optical system provide two side-by-side images of the same scene, to be combined and displayed at the operating/viewing station 18 as a stereoscopic image.
System 100 has only a single LWIR camera 11. Camera 11 may be a commercially available (or to be developed) thermal imaging camera. These thermographic cameras operate in the LWIR region, based on uncooled bolometer arrays as well as on cooled arrays. Camera 11 may provide still and/or motion images, and may have various resolution and frame rate options.
An optical system 12 provides two spatially separated fields of view to the camera 11. Optical system 12 has an active component that is operated in synchronization with the frame rate of the camera such that only one input field of view is visible to the camera at a time. This capability may be implemented in several forms, such as with a flipping or rotating mirror, a beam splitter with a mechanical shutter, or an electro-optical material operating as an attenuator that can vary its transmittance.
In the example of
System 100 further comprises an on-board transmitter 17, which provides a means to transmit the stereo video stream to an operator remote from the robot, as well as to receive commands from the operator to camera 11 for remote camera control. The streaming can involve transmission over an electrical cable, an optical fiber, or by wireless radio frequency transmissions. To keep data requirements low, the individual video streams may be multiplexed or interlaced.
An operating/viewing station 18 receives the video stream, and allows the operator to view the LWIR images such that the three dimensional stereo effect is visible. This can be accomplished by several means, the preferred being use of stereo viewing goggles, since this provides an immersive environment for the operator and frees the operator from other distractions. Other means for displaying three dimensional images include displays that utilize polarization or shuttered glasses, or by a conventional display that uses a bi-colored stereo anaglyph along with appropriately colored glasses. Further details pertaining to the split images and their processing are described below in connection with
A first pair of mirrors, camera mirrors 21a and 21b, provide top and bottom separation of the camera's field of view. These mirrors 21a and 21b are placed one above the other crosswise, at a predetermined angle. In the example of
A second pair of mirrors, input mirrors 22a and 22b, direct light from the forward direction (i.e., from the scene to be viewed by the robot) toward the camera mirrors 21a, 21b. The placement of input mirrors 22a and 22b determines the amount of stereo separation.
The complete image received by camera 11 has the two fields of view split between the top and bottom halves of the frame. Although implementing a side-by-side split of the image is also possible, a top-bottom spit is more useful for a robot driving application because the most important information is in a horizontal strip just below the horizon where objects directly in front of the robot reside.
In the example of
Although this implementation is rather large, the size could be reduced by use of a narrower field of view lens on the camera 11 combined with narrower, long focal length convex mirrors. These mirrors would act to increase the field of view of the camera while making a slightly more compact optical system.
The above-described stereo LWIR imaging provides immersive video for a robot operator to enhance their situational awareness and for improved understanding of the robot's surroundings. With stereo viewing, the shape of objects in an image is a more dominant factor than just their brightness or color. Since a user will have familiar visual depth cues to focus on, then object recognition is greatly improved over a single camera. The topography of the area through which the robot is navigating becomes readily apparent by use of stereo imagery allowing the operator to quickly select an optimal path. In addition, the use of stereo imaging appears to overcome some resolution limitations of the cameras since the operator has twice the amount of data to visualize their surroundings. The use of LWIR itself provides enhanced information. Different types of terrain (i.e. rocks, gavel, sand, etc.) are more readily distinguished. Objects with low visible contrast (such as plants and the ground) have high contrast in LWIR. Additionally, objects of interest to a robot operator such as people, animals, active electronics packages, voids, and disturbed ground can be distinguished by their LWIR emissions.
