The present disclosure relates to a system and method of controlling operation of a host device in real-time, using three-dimensional position and velocity information of at least one neighboring object. Mobile devices, including but not limited to motor vehicles, may be surrounded by other moving and non-moving objects. If the surrounding objects are positioned at an incline relative to the host device, it may be challenging to ascertain their respective trajectories.
A system and method of controlling operation of a host device in real-time includes an optical device operatively connected to the host device. The optical device is configured to obtain visual data, including one or more images of at least one object. The object is located at an incline, relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ). A radar device is operatively connected to the host device and configured to obtain radar data. The radar data includes a radial distance (r) of the object from the host device, the azimuth angle (θ), and a range rate (dr/dt).
A controller is operatively connected to the host device and includes a processor and tangible, non-transitory memory on which is recorded instructions. Execution of the instructions by the processor causes the controller to obtain respective bounding boxes around the object in the visual data. The controller is programmed to determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt).
The controller is programmed to determine a time-to-contact for the host device and the object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (vx, vy, vz). The operation of the host device is controlled based at least partially on the time-to-contact.
The host device may include an automatic braking module configured to decelerate the host device. Controlling operation of the host device may include sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold. Controlling operation of the host device may include activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.
The controller may be programmed to communicate with a remote module such that the remote module is not physically connected to the controller. The controller may be programmed to transfer a location coordinate of the host device to the remote module and receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate. In one embodiment, the controller is programmed to determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.
The system is configured to obtain three-dimensional position and velocity information for the object, by conjoining the radar data and the visual data. Because the position and velocity information is obtained in three dimensions, accuracy of the time-to-contact between the host device and the respective objects is significantly improved.
The controller may be programmed to match the respective bounding boxes in the visual data with the radar data. The controller may include an association module configured to perform the matching of the respective bounding boxes with the radar data. The controller is programmed to determine a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt), by combining the visual data and the radar data. The controller may include a Kalman filter module configured to perform the combining of the visual data and the radar data. A 3-D position vector (X, Y, Z) and a 3-D velocity vector (vx, vy, vz) are obtained from the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt), respectively.
The respective bounding boxes include a first bounding box B1 characterized by an initial position (u,v). The optical flow may be characterized by a first displacement change (Δu) of the respective bounding boxes in row coordinates and a second displacement change (Δv) of the respective bounding boxes in column coordinates, both over a predefined unit of time (Δt). The 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) include respective components determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt).
The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the disclosure when taken in connection with the accompanying drawings.
Referring to the drawings, wherein like reference numbers refer to like components,
Referring to
Referring to
Referring to
Referring to
Referring now to
In block 102A of
In block 102B of
In block 104, the controller C is programmed to obtain respective bounding boxes around the object 20 in each of the images in the visual data. Referring to
In block 104 of
Block 104 includes determining the change rate of the elevation angle (dϕ/dt) and the change rate of the azimuth angle (dθ/dt). The elevation angle (ϕ), azimuth angle (θ), the change rate of the elevation angle (dϕ/dt) and the change rate of the azimuth angle (dθ/dt) may be determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) .e., the projection of the radial distance (r) on the X-Y plane in the first coordinate system 24 and the predefined unit of time (Δt) such that:
(dϕ/dt)=Δu/(f*Δt);
(dθ/dt)=Δv/(f*Δt);
ϕ=inverse tangent(u/f); and
θ=inverse tangent(v/f).
In block 106 of
In block 108 of
In block 110 of
x=r sin ϕ cos θ;y=r sin θ sin ϕ;z=r cos ϕ;
vx=[(dr/dt)sin ϕ cos θ+r(dθ/dt)cos θ cos ϕ−r(dϕ/dt)sin θ sin ϕ];
vy=[(dr/dt)sin θ sin ϕ+r(dθ/dt)cos θ sin ϕ+r(dϕ/dt)sin θ cos ϕ];
vz=[(dr/dt)cos ϕ−r(dθ/dt)sin θ].
From block 110, the method 100 may proceed to block 111 (or block 114 directly, as shown by line 112). In block 111, the controller C is programmed to transmit a location coordinate (i.e., geophysical coordinates) of the host device 12 to the remote module 30 (see
From block 111, the method 100 proceeds to block 113, where the controller C is programmed to determine if the location assessment factor (LAF) is at or above a threshold value (e.g. 0.5). In one embodiment, the controller C is programmed to proceed to block 114 and determine the time-to-contact as long as the location assessment factor (LAF) is at or above the threshold value.
As noted above, the method 100 may proceed to block 114 directly from block 110 or from block 113. In block 114 of
t=√{square root over ((X/vx)2+(Y/vy)2+(Z/vz)2)}
In block 116 of
The controller C (and execution of the method 100) improves the functioning of the host device 12 by precisely obtaining time-to-contact between the host device 12 and the respective ones of the plurality of objects 20 and controlling operation of the host device 12 based at least partially on the time-to-contact. The controller C of
The controller C includes a computer-readable medium (also referred to as a processor-readable medium), including a non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory (DRAM), which may constitute a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of a computer. Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, other magnetic media, a CD-ROM, DVD, other optical media, punch cards, paper tape, other physical media with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, other memory chips or cartridges, or other media from which a computer can read.
Look-up tables, databases, data repositories or other data stores described herein may include various kinds of mechanisms for storing, accessing, and retrieving various kinds of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), etc. Each such data store may be included within a computing device employing a computer operating system such as one of those mentioned above, and may be accessed via a network in one or more of a variety of manners. A file system may be accessible from a computer operating system, and may include files stored in various formats. An RDBMS may employ the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures, such as the PL/SQL language mentioned above.
The detailed description and the drawings or FIGS. are supportive and descriptive of the disclosure, but the scope of the disclosure is defined solely by the claims. While some of the best modes and other embodiments for carrying out the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure defined in the appended claims. Furthermore, the embodiments shown in the drawings or the characteristics of various embodiments mentioned in the present description are not necessarily to be understood as embodiments independent of each other. Rather, it is possible that each of the characteristics described in one of the examples of an embodiment can be combined with one or a plurality of other desired characteristics from other embodiments, resulting in other embodiments not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the framework of the scope of the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
9429650 | Zeng | Aug 2016 | B2 |
9708004 | Weisswange | Jul 2017 | B2 |
20030210807 | Sato | Nov 2003 | A1 |
20100191391 | Zeng | Jul 2010 | A1 |
20120027258 | Uchida | Feb 2012 | A1 |
20120106786 | Shiraishi | May 2012 | A1 |
20120140076 | Rosenbaum | Jun 2012 | A1 |
20140037138 | Sato | Feb 2014 | A1 |
20150066348 | Baba | Mar 2015 | A1 |
20150234045 | Rosenblum | Aug 2015 | A1 |
20160054452 | Cosatto | Feb 2016 | A1 |
20170123429 | Levinson | May 2017 | A1 |
20180024562 | Bellaiche | Jan 2018 | A1 |
20190034742 | Gokan | Jan 2019 | A1 |
Number | Date | Country | |
---|---|---|---|
20180259637 A1 | Sep 2018 | US |