Patent Application
20200082561
This description relates to image processing and object detection in images, and particularly to mapping stationary objects detected in a plurality of images to geographic positions.
Digital electronic maps are widely used today for navigation, ride sharing, and video games, among other uses. While stand-alone map applications often include many of these functionalities, other applications can make use of electronic maps by calling a map server through an Application Programming Interface (API) on computing devices.
Some map applications process camera images in real time and detect objects present in the images. However, a single object detection in an individual image does not allow for an object to be placed with high precision on the map displayed by the map application. Existing techniques for mapping objects to geographic positions using one or more object detections are imprecise or cannot operate in real time as images are received and objects are detected on a client device. As a result, users of map applications see objects displayed in inaccurate positions on virtual maps or do not have access to object positions when they are most relevant. As a result, there is a need for map applications that can identify with high precision the geographic location of objects detected in processed images in real time.
A technique for mapping objects detected in a plurality of images to geographic positions is disclosed herein. In the embodiments discussed below the technique is implemented on a mobile computing device (e.g. a smart phone), although the method can be implemented on any client-side or server-side computing device, or on a combination thereof. The mobile computing device receives a plurality of images from a camera, e.g., a camera physically located within a vehicle. The mobile computing device inputs the images into a vision model loaded into a memory of the mobile computing device. The vision model is configured to generate a set of object detections (that is, data representing the classification and image location of detected objects) for objects appearing in the received images. The mobile computing device filters the set of object detections to comprise only detections of stationary objects (that is, inanimate objects that are not moving). The mobile computing device associates, with each object detection in the set of stationary object detections, camera information including at least the position of the camera when the image in which the object was detected was captured. Camera position information is obtained using the GPS receiver on the mobile computing device. The mobile computing device inputs the set of stationary object detections into an object mapping module. The object mapping module assigns each received stationary object detection to a group of object detections, where an object detection group consists of detections of the same distinct object in the environment. If no group exists for a received stationary object detection, then the object mapping module creates a new group and assigns the stationary object detection to the new group. When an object detection group receives a threshold number of object detections, the object mapping module localizes the position of the distinct object relative to the camera. The object mapping module converts the object position relative to the camera to a geographic position using the camera position information of the mobile computing device.
The geographic object positions output by the object mapping module can be used to display virtual content at geographically accurate locations. For example, the mobile computing device can display a virtual representation of a detected object on a digital map at the actual location where the object is on the Earth. In another example, the mobile computing device can display digital objects relative to the geographic position of the detected object on a digital map or in a live video feed on the mobile computing device.
A computer system 100 comprises components that are implemented at least partially by hardware at one or more computing devices, such as one or more hardware processors executing stored program instructions stored in one or more memories for performing the functions that are described herein. In other words, all functions described herein are intended to indicate operations that are performed using programming in a special-purpose computer or general-purpose computer, in various embodiments.
Server computer 105 may be any computing device, including but not limited to: servers, racks, workstations, personal computers, general purpose computers, laptops, Internet appliances, wireless devices, wired devices, multi-processor systems, mini-computers, and the like. Although
Server computer 105 is communicatively connected to database 120 and mobile computing device 145 through any kind of computer network using any combination of wired and wireless communication, including, but not limited to: a Local Area Network (LAN), a Wide Area Network (WAN), one or more internetworks such as the public Internet, or a company network. Server computer 105 may host or execute vision application 110, and may include other applications, software, and other executable instructions, such as database interface 117, to facilitate various aspects of embodiments described herein.
Database interface 117 is a programmatic interface such as JDBC or ODBC for communicating with database 120. Database interface 117 may communicate with any number of databases and any type of database, in any format. Database interface 117 may be a piece of custom software created by an entity associated with the vision application 110, or may be created by a third-party entity in part or in whole.
Database 120 is a data storage subsystem consisting of programs and data that is stored on any suitable storage device such as one or more hard disk drives, memories, or any other electronic digital data recording device configured to store data. Although database 120 is depicted as a single device in
Database 120 may be in any format, such as a relational database or a noSQL database. Database 120 is communicatively connected with server computer 105 through any kind of computer network using any combination of wired and wireless communication of the type previously described. Optionally, database 120 may be communicatively connected with other components, either directly or indirectly, such as one or more third party data suppliers. Generally, database 120 stores data related to electronic maps including, but not limited to: electronic map source data 125, electronic map data 130, telemetry data 135, and aggregated telemetry data 140. These datasets may be stored as columnar data in a relational database or as flat files, for example.
Electronic map source data 125 is raw digital map data that is obtained, downloaded or received from a variety of sources. The raw digital map data may include satellite images, digital street data, building data, place data or terrain data. Example sources include National Aeronautics and Space Administration (NASA), United States Geological Survey (USGS), and DigitalGlobe. Electronic map source data 125 may be updated at any suitable interval, and may be stored for any amount of time. Once obtained or received, electronic map source data 125 is used to generate electronic map data 130.
Electronic map data 130 is digital map data that is provided, either directly or indirectly, to client map applications, such as client map application 155, using an API. Electronic map data 130 is based on electronic map source data 125. Specifically, electronic map source data 125 is processed and organized as a plurality of vector tiles which may be subject to style data to impose different display styles. Electronic map data 130 may be updated at any suitable interval, and may include additional information beyond that derived from electronic map source data 125. For example, using aggregated telemetry data 140, discussed below, various additional information may be stored in the vector tiles, such as traffic patterns, turn restrictions, detours, common or popular routes, speed limits, new streets, and any other information related to electronic maps or the use of electronic maps.
Telemetry data 135 is digital data that is obtained or received from mobile computing devices via function calls that are included in a Software Development Kit (SDK) that application developers use to integrate and include electronic maps in applications. As indicated by the dotted lines, telemetry data 135 may be transiently stored, and is processed as discussed below before storage as aggregated telemetry data 140.
The telemetry data may include mobile device location information based on GPS signals. For example, telemetry data 135 may comprise one or more digitally stored events, in which each event comprises a plurality of event attribute values. Telemetry events may include: session start, map load, map pan, map zoom, map tilt or rotate, location report, speed and heading report, or a visit event including dwell time plus location. Telemetry event attributes may include latitude-longitude values for the then-current position of the mobile device, a session identifier, instance identifier, application identifier, device data, connectivity data, view data, and timestamp.
Aggregated telemetry data 140 is telemetry data 135 that has been processed using anonymization, chunking, filtering, or a combination thereof. Anonymization may include removing any data that identifies a specific mobile device or person. Chunking may include segmenting a continuous set of related telemetry data into different segments or chunks representing portions of travel along a route. For example, telemetry data may be collected during a drive from John's house to John's office. Chunking may break that continuous set of telemetry data into multiple chunks so that, rather than consisting of one continuous trace, John's trip may be stored as a trip from John's house to point A, a separate trip from point A to point B, and another separate trip from point B to John's office. Chunking may also remove or obscure start points, end points, or otherwise break telemetry data into any size. Filtering may remove inconsistent or irregular data, delete traces or trips that lack sufficient data points, or exclude any type or portion of data for any reason. Once processed, aggregated telemetry data 140 is stored in association with one or more tiles related to electronic map data 130. Aggregated telemetry data 140 may be stored for any amount of time, such as a day, a week, or more. Aggregated telemetry data 140 may be further processed or used by various applications or functions as needed.
Mobile computing device 145 is any mobile computing device, such as a laptop computer, hand-held computer, wearable computer, cellular or mobile phone, portable digital assistant (PDA), or tablet computer. Although a single mobile computing device is depicted in
Mobile computing device 145 is communicatively coupled to GPS satellite 160 using GPS receiver 150. GPS receiver 150 is a receiver used by mobile computing device 145 to receive signals from GPS satellite 160, which broadly represents three or more satellites from which the mobile computing device may receive signals for resolution into a latitude-longitude position via triangulation calculations.
Mobile computing device 145 also includes wireless network interface 159 which is used by the mobile computing device to communicate wirelessly with other devices. In particular, wireless network interface 159 is used to establish wireless network connection 165 to server computer 105. Wireless network interface 159 may use WiFi, WiMAX, Bluetooth, ZigBee, cellular standards or others.
Mobile computing device 145 also includes other hardware elements, such as one or more input devices, memory, processors, and the like, which are not depicted in
Mobile computing device 145 also includes a camera device 147. The camera 147 may be external, but connected, to the mobile computing device 145. Alternatively, the camera 147 may be an integrated component of the mobile computing device 145. Camera 147 functionality may include the capturing of infrared and visible light.
Mobile computing device 145 may include a client map application 155 which is software that displays, uses, supports, or otherwise provides electronic mapping functionality as part of the application or software. Client map application 155 may be any type of application, such as a taxi service, a video game, a chat client, a food delivery application, etc. In one embodiment, client map application 155 obtains electronic mapping functions through SDK 157, which may implement functional calls, callbacks, methods or other programmatic means for contacting the server computer to obtain digital map tiles, layer data, or other data that can form the basis of visually rendering a map as part of the application. In general, SDK 157 is a software development kit that allows developers to implement electronic mapping without having to design all of the components from scratch. For example, SDK 157 may be downloaded from the Internet by developers, and subsequently incorporated into an application which is later used by individual users.
The trained vision model 115 receives images from the camera 147. In one embodiment, the client map application 155 may also receive processed images from the trained vision model 115. In one embodiment, the trained vision model 115 is configured to output sets of object detections.
The object mapping module 167 receives object detections from the trained vision model 115. In one embodiment, the object mapping module 167 also receives a position of the camera 147 included with each object detection, where the camera position corresponds to the time the image containing the object detection was captured by the camera. In one embodiment, the object mapping module 167 is configured to output the geographic position of each detected object. As used herein, the term “geographic position” refers to a location on the Earth's surface. For example, a geographic position may be represented using longitude-latitude values.
In server computer 105, the vision application 110 provides the API 112 that may be accessed, for example, by client map application 155 using SDK 157 to provide electronic mapping to client map application 155. Specifically, the vision application 110 comprises program instructions that are programmed or configured to perform a variety of backend functions needed for electronic mapping including, but not limited to: sending electronic map data to mobile computing devices, receiving telemetry data 135 from mobile computing devices, processing telemetry data to generate aggregated telemetry data 140, receiving electronic map source data 125 from data providers, processing electronic map source data 125 to generate electronic map data 130, and any other aspects of embodiments described herein.
As shown in
The object mapping module 167 receives object detections including respective camera 147 positions as input. For each distinct object in the environment about the vehicle 175, the object mapping module outputs a geographic position of the object. In one embodiment, after the client map application 155 receives the geographic position of a distinct object in the environment it positions the object on a map displayed on the mobile computing device screen by the client map application. In the same or different embodiment, after processing by the client map application 155, the live camera view or some processed view thereof is displayed on the mobile computing device screen for the user's view. In the same or a different embodiment, computer code associated with a software application loaded on the mobile computing device 145 (e.g., the client map application 155) alerts the user regarding detected objects mapped to geographic positions and positioned on a map in the client map application, examples of which include but are not limited to road signs, cross-walks, and traffic lights. In the same or a different embodiment, the output of the object mapping module 167 is used to provide new or updated map information to the server 105, including locations of road signs, cross-walks, and traffic lights.
Although in the embodiments described herein the operating environment for the object mapping module is on a mobile computing device, one skilled in the art will readily recognize from the following discussion that alternative embodiments are possible. For example, object mapping module may operate on server 105 where object detections have been aggregated from one or more mobile computing devices.
Although there are a number of model architectures that may function adequately for performing detection and image segmentation tasks on a set of images, generally these model architectures are designed for use with traditional desktop or cloud computing resources, both in terms of processor computation ability, and also in that they have wired connection to electrical power. Mobile computing devices, by contrast, are limited in both regards. As such, model architectures that require a great deal of electrical power or compute ability are infeasible for use with mobile computing devices.
Particularly in this context, the goal of the trained vision model 115 is to run continuously on the mobile computing device 145 as a driver operates a vehicle traveling from one destination to another, while consuming as little compute ability and power as possible, while also achieving desired object detection and segmentation on the images processed by the trained vision model 115. The methods taught in pending U.S. patent application Ser. No. 16/354,108, entitled “Low Power Consumption Deep Neural Network for Simultaneous Object Detection and Semantic Segmentation in Images on a Mobile Computing Device,” filed on Mar. 14, 2019, disclose an embodiment of a trained vision model working in this desired context.
If an object detection group 537 receives 535 a threshold number of object detections required by the object mapping module 167, the object detection assignment module 530 inputs the object detection group 537 into the object localization module 540. The object localization module 540 processes the plurality of object detections in a group corresponding to a distinct object to compute a position of the distinct object relative to the camera 147. The relative object position 545 describes the real-world position of the distinct object in relation to the camera 147 at various real-world positions. In one embodiment, the object detection includes camera parameters which describe attributes of camera 147 such as the type of lens and the lens focal distance. In the same or different embodiment, the object localization 540 module has access to known sizes for specific object classification labels. In some embodiments, these known sizes are provided by a size range. For example, a stop sign classification may have a corresponding known size of 1.8-2 m height, 0.5-0.7 m width, and 0.2-0.3 m length. Several embodiments of the object localization module 540 are discussed in section VII, below.
The geographic position conversion module 550 receives each relative object position 545 from the object localization module 540 as input and converts each relative object position 545 to a geographic position 560. In one embodiment, the geographic conversion module 550 uses a known geographic camera position derived from the geolocation of the mobile computing device 145 to perform the conversion, e.g., by the coordinate components of the relative object position to those of the geographic camera position.
Object localization using grouped object detections relies on known estimations of positions, orientations, and sizes of objects in the real world. In various embodiments, these estimated values include a camera position, a camera orientation, and an object size associated with an object classification label. The accuracy of the object positions relative to the camera 147 output by the object localization module 540 depends on the accuracy of these estimated values. As such, there are a number of possible object localization techniques that vary in which known estimated values are used and in the accuracy of the output object position. In particular, alternative techniques employed by different embodiments of the object localization module 540 are now discussed, including (A) a system of linear equations for pixels, (B) ray intersection, and (C) Gaussian fusion.
VII.A. Solving System of Linear Equations for Pixels
In
VII.B. Intersecting Ray Projections
In
VII.C. Fusing Probability Distributions
In one embodiment, the location and size of the probability distribution along a given ray is determined based on the pixel coordinates of the bounding box and the known object size. In the same or different embodiment, the position and boundary of the probability distribution in the plane parallel to the ray are based on the position and orientation of the camera 147. In the same or different embodiment, the probability distribution is represented as a multivariate normal distribution (i.e. a gaussian distribution) over two variables. For example, the two variables can be coordinates on the 2-dimensional plane parallel to the ray which are relative to a specific point on the ray where the distribution is centered. In the same or different embodiment, the probability distributions are fused by multiplying each distribution together and normalizing the resulting distribution.
In
The detected object geographic positions may be used in a number of contexts.
VIII.A. Adding Objects to Map
Each detected object geographic position identifies the real-world location of an object on the Earth's surface. In some embodiments, the client map application 155 uses the geographic position of a detected object and relevant information from corresponding object detections to display a representation of the object on a digital map of the Earth's surface. The client map application 155 can represent the object on a digital map in any format applicable to the functions of the particular map application. For example, if the object mapping module 167 outputs the geographic position of a speed-limit sign, the client map application 155 could display the image of a speed-limit sign with a corresponding speed on the digital map at a map position corresponding to the geographic position.
In some embodiments, the client map application 155 uses the geographic position of a detected object and relevant information from corresponding object detections to display virtual objects in a live video feed displayed by the mobile computing device 145. For example, a client map application 155 may display a virtual stop-sign on the live video feed of mobile computing device 145 to highlight the real-world object to a user, including adding visual emphasis to the virtual stop-sign such as a colored highlight around its outline or on its surface.
VIII.B. Updating Server Map Information
Object geographic positions can be sent to the map server by the mobile computing device 145. Object geographic positions previously added to the map server may be incorrect due to inaccurate camera position or orientation information, too few object detections considered, object detections incorrectly assigned to an object detection group, inaccurate geolocation of the mobile computing device due to phone movement, or the like. To address this, geographic positions of road-related objects such as road signs can be added/updated/deleted on the map server. The object mapping module 167 outputs a set of geographic object positions obtained from object detections provided by the trained vision model 115. The mobile computing device 145 sends detected object information including object geographic positions to the server computer 105, and the server computer 105 receives 745 data describing the detected objects from the mobile computing device 145. The server computer 105 updates 755 the stored map with the detected object geographic positions output by the object mapping module 167.
In one embodiment, the set of live images 425 includes multiple object detections of an object relevant to driving decisions. The trained vision model 115 classifies the object in the images in which it appears. The object mapping module 167 determines a geographic position of the object. The server computer 105 checks the existing map repository for an object of the same type as the classified object near the determined geographic position. If there are discrepancies between the geographic position of the client map application's object and that in the existing map repository, the server computer 105 adjusts the geographic position in the existing map repository to reflect the geographic position of the client device and updates stored database of map information in real time to reflect the geographic position of the client map application's object.
VIII.C. Real Time Object Notifications
In response to a number of rules stored in the memory of the mobile computing device 145, the client map application 155 may notify users about the location of mapped objects either by visual notification or audio alert via the screen or speaker of the mobile computing device 145, respectively. Notifications may be directed at aiding user navigation, encouraging safe driving habits if operating from a vehicle, or alerting users to objects in the environment that may be relevant. For example, a user may be driving a vehicle approaching a geographically positioned stop-sign at a speed indicating the user may not come to a complete stop. In this case, the client map application 155 may alert the user of the stop sign's position within time for the user to safely bring the vehicle to a stop. Rules triggering user notifications may depend on data, such as GPS location or type of road sign, collected from the mobile computing device 145, on which the client map application 155 is running.
In one embodiment, the client map application 155 or object mapping module 167 determines that one or more of the objects mapped to geographic positions by the object mapping module 167 are hazardous. The client map application 155 may automatically warn the user when their vehicle is within a detected distance from the hazard. For example, the client map application 155 may detection and position a stationary object in the lane in which a user is driving a vehicle. In this case, the client map application may alert the user to the stationary object so that the user can change lanes or otherwise avoid the object.
In another embodiment, users of the client map application 155 may set the rules that result in notifications. For example, a user may choose to be notified when the object mapping module 167 determines the geographic position of an object identified as a stop sign ahead of the vehicle.
Example computer system 900 also includes a main memory 906, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 902 for storing information and instructions to be executed by processor 904. Main memory 906 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 904. Such instructions, when stored in non-transitory storage media accessible to processor 904, render computer system 900 into a special-purpose machine that is customized to perform the operations specified in the instructions.
Computer system 900 further includes a read only memory (ROM) 908 or other static storage device coupled to bus 902 for storing static information and instructions for processor 904. A storage device 910, such as a magnetic disk or optical disk, is provided and coupled to bus 902 for storing information and instructions.
Computer system 900 may be coupled via bus 902 to a display 912, such as a LCD screen, LED screen, or touch screen, for displaying information to a computer user. An input device 914, which may include alphanumeric and other keys, buttons, a mouse, a touchscreen, or other input elements is coupled to bus 902 for communicating information and command selections to processor 904. In some embodiments, the computer system 900 may also include a cursor control 916, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 904 and for controlling cursor movement on display 912. The cursor control 916 typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
Computer system 900 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and program logic which in combination with the computer system causes or programs computer system 900 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 900 in response to processor 904 executing one or more sequences of one or more instructions contained in main memory 906. Such instructions may be read into main memory 906 from another storage medium, such as storage device 910. Execution of the sequences of instructions contained in main memory 906 causes processor 904 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “storage media” as used herein refers to any non-transitory media that store data and instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 910. Volatile media includes dynamic memory, such as main memory 906. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 902. Transmission media can also take the form of acoustic, radio, or light waves, such as those generated during radio-wave and infra-red data communications, such as WI-Fl, 3G, 4G, BLUETOOTH, or wireless communications following any other wireless networking standard.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 904 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 900 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 902. Bus 902 carries the data to main memory 906, from which processor 904 retrieves and executes the instructions. The instructions received by main memory 906 may optionally be stored on storage device 910 either before or after execution by processor 904.
Computer system 900 also includes a communication interface 918 coupled to bus 902. Communication interface 918 provides a two-way data communication coupling to a network link 920 that is connected to a local network 922. For example, communication interface 918 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 918 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 918 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 920 typically provides data communication through one or more networks to other data devices. For example, network link 920 may provide a connection through local network 922 to a host computer 924 or to data equipment operated by an Internet Service Provider (ISP) 926. ISP 926 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 928. Local network 922 and Internet 928 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 920 and through communication interface 918, which carry the digital data to and from computer system 900, are example forms of transmission media.
Computer system 900 can send messages and receive data, including program code, through the network(s), network link 920 and communication interface 918. In the Internet example, a server 930 might transmit a requested code for an application program through Internet 928, ISP 926, local network 922 and communication interface 918. The received code may be executed by processor 904 as it is received, and stored in storage device 910, or other non-volatile storage for later execution.
This application claims the benefit of Provisional Application No. 62/729,401 (Atty. Docket #33858-41353), filed on Sep. 10, 2018, and of Provisional Application No. 62/834,370 (Atty. Docket #33858-43377), filed on Apr. 15, 2019, both of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62729401 | Sep 2018 | US | |
| 62834370 | Apr 2019 | US |
