Patent Application
20180322653
The present invention relates to positioning systems, and, more particularly, to an apparatus and method for providing vehicular positioning within a particular environment using multiple optical cameras having contiguous fields of view in combination with coded markers.
Positioning systems for objects such as moving vehicles come in a variety of well-known configurations. For example, a global navigation satellite system (GNSS) such as the well-known US Global Positioning System (GPS), the Russian GLONASS or European GALILEO, utilize radio frequency signals and other location information received from satellites for providing outdoor location and positioning with a high degree of accuracy. However, these satellite signals become weak and attenuated for indoor applications due to their line-of-sight requirements and GPS devices, for example, have a difficult time locking onto such signals when operating indoors. Further, for example, construction vehicles operating inside a large building or domed stadium structure (on the scale of 100 meters wide and tens of meters high) require the use of robotic total stations in performing construction work and these large-scale indoor structures are, in terms of using GNSS signals, susceptible to multi-path propagation effects, power loss as GNSS signals pass through construction materials, and electromagnetic and audio interference thereby making indoor positioning a challenge using GNSS signals.
As such, a variety of positioning techniques exist to facilitate indoor positioning and alleviate some of the above-detailed disadvantages with respect to using GNSS signals for indoor applications. For example, certain techniques utilizing received signal strength indication (RSSI) from Wi-Fi and Bluetooth® wireless access points have been identified, however, complex indoor environments cause these types of radio waves to propagate in dynamic and unpredictable ways thereby limiting the overall positioning accuracy using RSSI.
Another indoor positioning alternative utilizes ultrasonic sonar which transmits acoustic waves to microphones in order to approximate indoor positions. This alternative technique operates at lower frequencies than Wi-Fi based systems and attenuate significantly when passing through walls, for example, thereby potentially increasing positioning accuracy.
A further positioning technique uses optical signals, either visible or infrared, which tend to have better accuracy than RSSI and/or ultrasonic sonar given that optical signals are highly directional and do not penetrate solid objects. Such optical techniques may use well-known light identification detection and ranging (LIDAR) which is a remote sensing method, similar to radar, that uses a laser pulse transmitted by a transmitter and the light particles (photons) are scattered back to a receiver and examined for position determination. Markers and landmarks may also be used where markers are active if they contain a transmitter emitting light or passive if the reflected ambient light is utilized, and/or a landmark position can be predetermined (or determined in real-time using the well-known simultaneous localization and map building (SLAM) technique). Nevertheless, the directionality of such optical signals limits their potential reliability in certain operating environments given the necessity to properly align the receiver and transmitter associated with the optical signals.
Therefore, a need exists for an improved technique for indoor vehicular positioning within large-scale structures.
In accordance with various embodiments, a vehicular positioning apparatus and method is provided that utilizes multiple optical cameras having contiguous fields of view for reading coded markers having pre-determined positions for determining the position of vehicle inside a particular environment (e.g., a structure) with a high degree of accuracy. Unlike conventional indoor vehicular positioning techniques based on total station or laser sensors, the disclosed embodiments provide for the direct installation and use of the positioning apparatus on the vehicle (as opposed to externally positioned devices) with a limited number of coded markers to determine the vehicle's position and deliver millimeter level accuracy which is sufficient for a majority of industrial grading and construction applications, for example. This configuration also reduces overall mechanical complexity, reduces manufacturing costs, and improves antitheft characteristics.
More particularly, in accordance with an embodiment, the position of an object (e.g., a moving vehicle) within a particular environment is determined by acquiring one set of images, in real-time (or substantially real-time) or at some defined time intervals, of at least three (3) visual markers located within the structure in which the vehicle is operating inside, as taken from the camera(s) of the positioning apparatus mounted to the object (e.g., the vehicle) for which the positioning determination is desired. In accordance with an embodiment, the set of visual markers may be any type of target marker that is of sufficient contrast and shape that such marker is readily and reliably detectable by the image acquisition module (e.g., a camera). Such visual markers may include checker-board markers, quick response (QR)-codes, barcodes (color or greyscale), surveying prisms, and contrasting circular markers, to name just a few. Each visual marker is located at a fixed (i.e., known) position within an indoor environment (e.g., within an arena and mounted on a wall or support column or post within such structure) and, in addition to the known location, the geometric features of the marker are also known (e.g., orientation and size of each marker). In accordance with the embodiment, the geometric features of each marker captured in a particular image are extracted and the indoor position of the vehicle is determined based on the extracted marker information.
When a visual marker contains multiple visible elements (such as cells in a checker-board configuration), the disclosed method can use the visible elements to further improve the accuracy of positioning. This is accomplished by matching the observed structure of visible elements with an expected structure of the visible elements, calculating the deviation (e.g., root mean square deviation) between the observed and expected structures, and updating the expectation (i.e., the expected structure) by minimizing the deviation. In order to minimize the deviation, for example, a non-linear minimization can be used. In this way, the disclosed embodiments allow for determining marker position at sub-pixel accuracy, and marker orientation can be estimated and used for precise vehicle positioning.
Another useful way to employ the information coded into the visual markers is to use a select method (many well understood methods of visual coding of information can be utilized effectively) of visual coding (e.g., by adding additional cells coding bits by black/white color, by differently alternating the colors in the checker-board, by using a variety of different colors, and/or by using QR-codes) to code the number (identifier) of marker, its coordinates and other information, which, having been detected and decoded by the positioning apparatus, will allow for to automatically corresponding the detected mark to its coordinates.
In accordance with the embodiments herein, the determination of a precise indoor position of an object (e.g., a vehicle) is facilitated by a combination of data collected from an optical system providing images of visual markers within the object's indoor environment, and data collected from inertial sensors and/or other sensor types. In this way, precise indoor positioning (e.g., within centimeters or millimeters) is achievable in large-scale indoor environments (e.g., hundreds of square meters) in the event the visual markers are not observable and images cannot be acquired for a particular time period. In addition, the position determination using only the mean of the optical system will have certain latency (delay) and frequency caused by the necessary time of taking and processing of the images, for which the delay and/or frequency might not be sufficient for the purpose of operating the vehicle, and in such case, an inertial sensor might be a source of positioning information during a short interval between two consequent position measurements with the optical sensor.
These and other advantages of the embodiments will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.
In accordance with various embodiments, an indoor vehicular positioning system is provided that utilizes multiple optical cameras having contiguous fields of view for reading coded markers having pre-determined positions for determining the position of vehicle within an indoor environment with a high degree of accuracy.
Processor 125 (and/or any of the processing modules described herein) may be any suitable processor, computing module, data processing device, or combination of the same. Processor 125 is communicatively coupled with memory 140 and with data module 130 (and indirectly with storage device 135 through data module 130) for storing computer-readable program instructions that provide the governing logic for controlling processor 125 (and/or other processing modules herein) to execute computer-readable program instructions stored in storage device 1.35 or other computer readable medium. Processor 125 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of positioning apparatus 100. Processor 125 may comprise one or more central processing units (CPUs), for example. Processor 125 and memory 140 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs).
Storage device 135 and memory 140 may each comprise a tangible non-transitory computer readable storage medium. Storage device 135 and memory 140, may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices.
Communications module 145 is a module which may include input means, such as a keypad, touchscreen, haptic controls, voice-recognition means and the like, used to identify user command input, for example. Communications module 145 also includes an output means, such as to a display, or to an external control unit which operates the vehicle, or a monitor to output images from image acquisition module 105, indicate an input command from a user, and display results of the calculation of the indoor position of vehicle, for example. Such communications may be exchanged by any number of well-known communications protocols such as Ethernet, Universal Serial Bus (USB), or Controller Area Network (CAN) bus, to name just a few. Communications module 145 may also include a user input/output device interface, which interfaces with a computer (not shown) with the user input/output. Examples of the user input/output device may include a keyboard, a mouse, a touchscreen, a microphone (for voice-activated input), and a local access terminal. The communications module 145 may further include a display interface, which interfaces the computer with a display for the display of data and images.
Communications module 145 may further include a GNSS receiver interface (not shown), which interfaces positioning apparatus 100 with a GNSS receiver (not shown) for receiving signals and measurements from the GNSS receiver (e.g., via communications signals 180) through the GNSS receiver interface. Further, communications module 145 may further include a total station interface, which interfaces positioning apparatus 100 with a total station operating within the indoor environment.
Image acquisition module 105 and markers detection module 115 are configured so each receives image input 150 generated from one or more image capture devices such as any type of camera, with one camera type preferably having a low distortion lens and high temperature stability in order to maintain image accuracy even in the midst of a harsh mechanical and/or thermal environment (e.g., a mining operation or tunnel). Such cameras (for example, camera unit 320 as depicted in
As noted above, in accordance with an embodiment, the indoor position of a vehicle is determined by simultaneously acquiring a set of images, in real-time (or substantially real-time) or at certain defined time intervals, of three (3) or more markers using multiple image capture devices (e.g., cameras) with contiguous fields of view, where the position and orientation of each camera with respect to the vehicle is known (and, illustratively, stored for further access and use). Illustratively, the set of images captures images of a set of visual markers located within the indoor environment (e.g., a building) in which the vehicle is operating inside. In accordance with an embodiment, the set of visual markers, the position of each marker being defined by and known within the applicable coordinate system of the worksite or applicable area for which positioning is being undertaken, may be any type of target marker that is of sufficient contrast and shape that such marker is readily and reliably detectable by the image capture device (e.g., a camera). In this way, by using markers consisting of varying and/or multiple elements the overall accuracy of estimating marker position in an image is improved given the enhanced ability to detect varying/multiple elements from the marker (and their corresponding positions). For example, by using an optimizalion technique and solving therefore, an estimate of a probable position on the image for the center of the marker (or other points, as required) may be determined given the measured points of all the marker elements. Such visual markers may include checker-board markers, quick response (QR)-codes, barcodes (color or greyscale), surveying prisms, and contrasting circular markers, to name just a few.
As will be noted, the characteristics of the visual markers herein (such as visual marker 200 which is one of many possibilities) are that the marker must be of sufficient contrast and shape for detection, easily and reliably, by image acquisition module 105, markers detection module 115, and the associated cameras (e.g., cameras 330-1 through 330-N) capturing the images of the visual markers and producing image input 150. In addition to the checkerboard configuration of visual marker 200, the visual markers may also include quick response (QR)-codes, barcodes (color or greyscale), surveying prisms, and contrasting circular markers, to name just a few. As illustrated in
As noted above, in accordance with various embodiments, an indoor vehicular positioning system is provided that utilizes multiple optical cameras having contiguous fields of view for reading coded markers having pre-determined positions for determining the position of vehicles inside an indoor environment with a high degree of accuracy. In particular, image acquisition module 105 receives images from one or more cameras associated with the plurality of visual markers (i.e., image input 150) Image acquisition module 105 is further configured to optionally define regions of interest around a visual marker's predicated position. As will be appreciated, a number of methods for predicting marker position can be utilized, for example, a vehicle movement model or optical flow algorithm. Alternatively, as noted above, optional inertial sensor(s) can be utilized to define velocity and direction of vehicle movement. Filtering module 110 analyzes image input 150 and may normalize image qualities such as brightness and contrast and send image parameters to image acquisition module 105 for improving the overall image quality received from the cameras via image input 150.
Markers detection module 115 receives image input 150 as well as information associated with the configuration of the plurality of visual markers and their potential location and size as determined by prediction module 175. In particular, prediction module 175 using known coordinates associated with the plurality of visual markers will predict a position in the frame of the respective marker from the camera image taken and the size of the marker. The markers detection module 115 will extract any encoded information (e.g., encoded information 230) from image input 150, and using image input 150 in combination with the prediction information received from predication module 175, will search for a visual marker image with the most likely positions. That is, the prediction module will define an area where the vehicle may appear in a given amount of time, and every point in the defined area may be projected to the camera(s). Thus, all points in the defined area produce a number of regions, in the camera(s), where the markers may appear in a certain time interval, and these regions may be utilized to optimize and/or accelerate the marker detection operations. In this way, prediction module 175 has prediction ability as a function of the movement model and/or one or more inertial measurements (e.g., processed by inertial sensors module 165) that is focused on the marker position(s) in the corresponding regions. As such, this translates into an overall decrease in the time to make one or more measurements thereby increasing the frequency of measurements and reducing associated time delays while maintaining overall accuracy of the vehicular positioning at higher vehicle speeds.
The predictions made by prediction module 175 may also be utilized by image acquisition module 105 to define, expand or restrict, for example, the areas from which image input 150 is desired within the indoor environment. Image input 150 is also utilized by markers detection module 115 to identify the specific marker ID (e.g., marker ID 220) associated with a particular visual marker (e.g., visual marker 200).
The search results from markers detection module 115 are provided to coordinates and position module 120 which will calculate the horizontal and vertical angle to each marker from the corresponding camera based on the location of the marker within the respective image. As will be appreciated, improved accuracy may be obtained in further embodiments using the location of elements of the marker (e.g., cells, corners, or parts of the QR-code) within the image. The calculated angles, from image input 150, are used in combination with known data of the indoor environment, with position and orientation of each camera with respect to the vehicle, and data from other sensors for determining a precise indoor position using well-known techniques in the field of geodesy (e.g., 3D resection).
The data from other sensors is sensory data 170 which is collected by and through inertial sensors module 165. That is, the determination of a precise indoor position of an object (e.g., a vehicle) is facilitated by a combination of data collected from an optical system providing images of visual markers within the object's indoor environment, and data collected from inertial (or other) sensors, in the event the visual markers are unobservable (or have reduced visibility) by the cameras capturing image input data 150 for provision to image acquisition module 105, or to provide the measurement of position in between two consequent measurements with optical means, in order to reduce the delay and/or increase the frequency of the position measurement as required. These sensors may include a wide variety of sensors (e.g., a gyro sensor and/or acceleration sensor). Data module 130 updates the data from image acquisition module 105 and inertial sensors module 165, and supports the execution of the overall motion model employed by positioning apparatus 100 in a well-known fashion. For example, in accordance with an embodiment the motion model can be an approximation of the movement of a particular vehicle with its given limitations for speed and acceleration, and a Kalman filter or variation could be used to integrate the position or angle measurements from the optical system with the acceleration measurements from the inertial sensors. As such, data module 130 can provide predictions of the position at the time that a camera provides a next image (via image input 150) to image acquisition module 105, by integrating the acceleration measurements from the inertial system and/or extrapolating the previous positions. Data module 130 may also receive a variety of information (e.g., information 160) from sources external to positioning apparatus 100 such as the visual marker information (see, e.g.,
In an embodiment for combined indoor/outdoor or strictly outdoor applications, as shown in
As also shown in
The camera unit 320, therefore, facilitates the capturing of individual images of the visual markers, i.e., visual marker 340-1, 340-2, 340-3, 340-4, and 330-N (as shown each such visual marker is a checkerboard marker configured substantially the same as visual marker 200, as detailed above). The individual markers are referenced to the geodesic grid in a well-known fashion for purposes of using the individual coordinates associated with visual marker 340-1, 340-2, 340-3, 340-4, and 340-N to building a 3D coordinate system model associated with indoor environment 395 in which vehicle 310 is currently located.
In accordance with the embodiments herein, images of at least three (3) different visual markers are necessary but may be desirable to capture images of more than 3 different visual markers captured by camera unit 320 in order to have a certain redundancy in measurements to facilitate outlier detection, and/or to improve the accuracy of the positioning by averaging or other similar mathematical method. As such, this provides a certain self-diagnostic or correction feature (e.g., incorporated within markers detection module 115, or coordinates and positioning module 120, or as a stand-alone module) in accordance with further embodiments that is useful either at the time of initial system installation and/or during the system's useful life to ensure on-going accuracy and proper positioning is maintained in accordance with various embodiments. More particularly, utilizing more than 3 different visual markers provides for a certain measurement redundancy thereby improving the ability to detect and/or correct for possible outlier(s) in positions/coordinates (i.e., erroneous positions/coordinates) in situations, for example, where markers may have been moved from a prior established position, or their respective coordinates cannot be measured or recorded correctly at a particular time. In such cases, the erroneous coordinates associated with the marker may be detected, and new coordinates are estimated for the associated set of coordinates by observing this marker from at least two different positions together with at least 3 other markers having known (i.e., correct) coordinate positions. This establishes, through the newly estimated coordinates, a corrected position for the previous outlier marker.
It should be noted that for clarity of explanation, the illustrative embodiments described herein may be presented as comprising individual functional blocks or combinations of functional blocks. The functions these blocks represent may be provided through the use of either dedicated or shared hardware, including, but not limited to, hardware capable of executing software. Illustrative embodiments may comprise digital signal processor (“DSP”) hardware and/or software performing the operation described herein. Thus, for example, it will be appreciated by those skilled in the art that the block diagrams herein represent conceptual views of illustrative functions, operations and/or circuitry of the principles described in the various embodiments herein. Similarly, it will be appreciated that any flowcharts, flow diagrams, state transition diagrams, pseudo code, program code and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer, machine or processor, whether or not such computer, machine or processor is explicitly shown. One skilled in the art will recognize that an implementation of an actual computer or computer system may have other structures and may contain other components as well, and that a high level representation of some of the components of such a computer is for illustrative purposes.
The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/RU2016/000589 | 8/31/2016 | WO | 00 |
