The present invention relates generally to location determination, and more particularly to determining the geolocation of objects in an image.
Cameras are often equipped with location determination hardware, such as a GPS receiver, which can determine the location of a camera. After an image is captured, information concerning the location of the camera at the time the image was captured can be stored with image data, allowing a user to know the location of the camera when the image was captured.
Although cameras with location determination hardware can produce image data including location information for association with images, the location information indicates the location of the camera and not the location of objects in an image.
In one embodiment, a method for determining a location of an object depicted in an image begins with receiving camera location data and object distance data associated with an image. The location of the object depicted in the image is determined based on the camera location and object distance data. In one embodiment, camera orientation data is also received and the location of the object is additionally based on the orientation data. Camera orientation data can include a direction the camera is pointing when the image is captured (i.e., azimuth) and an angle from horizontal the camera is pointing (i.e., elevation angle). The distance of the object from the camera is determined, in one embodiment, based on distance to subject data and, in another embodiment, based on focal length data. The direction of the object with respect to the camera is determined, in one embodiment, based on camera orientation data.
An apparatus for determining the location of an object depicted in an image is also disclosed.
These and other advantages of the invention will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.
The present disclosure pertains to a method and apparatus in which images captured using a device, such as a digital camera, are analyzed to determine the location of objects depicted in the image. In contrast, data typically associated with an image pertains only to the location of the image capturing device (e.g., camera) at the time the image was taken and not the location of objects in the image. The location of an object depicted in an image, according to one embodiment, is determined based on a location of the camera used to capture the image at the time the image was captured, object distance data, and camera orientation data.
Camera 102 can be used to take a picture of object 104, in this case, a beach ball. After an image of the beach ball is taken, it is saved in a memory of camera 102. A captured image is stored in camera memory with associated image data.
Images can be generated and stored in various formats such as JPEG (Joint Photographic Experts Group), TIFF(Tagged Image File Format), and raw image format. Image data (also referred to as image metadata) contains information about an image and can be stored in a specific format (e.g., EXIF (Exchangeable Image File Format)).
Images taken using camera 102, together with associated image data, can be uploaded to a device such as computer 106. Alternatively, images may be uploaded to another device, such as image server 110, via network 108. Image server 110, in one embodiment, is in communication with image database 112 which may be used to store images and associated image data.
Image data may be obtained using various hardware. For example, location 208 identifies the location of camera 102 when an image is taken and, in one embodiment, is obtained using a GPS receiver built into camera 102. Location information can alternatively be determined using other types of hardware for determining location. Location information can be stored in location 208 in various formats. As shown in
Focal length 210 indicates the focal length of the camera optics at the time an image was captured.
Information concerning focal length, in one embodiment, is determined by camera hardware. In one embodiment, a camera has a non-removable lens and focal length is determined based on orientation of the camera optics (i.e., where the focus has been set). For example, the distance between the lens and image sensor of the camera. In one embodiment, camera 102 can be equipped with one of several different lenses. In this embodiment, the particular lens attached to camera 102 may be automatically determined by camera 102 or manually entered by a user.
Returning to
Distance to subject 212 can also be obtained using other hardware included in camera 102 such as an infrared based or sound based range sensor included in many cameras for autofocus. Distance to subject 212 may also be calculated based on focal length information together with optics information pertaining to the particular camera used, image sensor size information, where the focus is set, etc. In one embodiment, data from an EXIF file associated with an image (i.e., distance to subject information) can be used to populate distance to subject 212.
Azimuth 214 indicates an angular orientation of camera 102 with respect to a reference. Azimuth 214 indicates an angular orientation of camera 102 with respect to north, which in this embodiment, is represented by zero. In one embodiment, azimuth 214 is determined by camera hardware such as a solid state compass. Alternatively, azimuth 214 can be determined using a GPS receiver alone or in conjunction with accelerometers. Elevation angle 216 (also known as altitude) indicates the orientation of camera 102 with respect to horizontal. Elevation angle 216 shown in
It should be noted that values for image data 204-216 associated with a particular image ID 202 pertain to camera location and orientation at the time the image identified by image ID 202 was taken.
Images and image data stored on camera 102 can be uploaded to computer 106 via a wired connector (e.g., USB) or wirelessly. Images and image data stored on camera 102 can alternatively be uploaded in a similar manner to image server 110 via network 108. In addition, images and image data can be transferred from computer 106 to image server 110 via network 108.
Images and image data uploaded to computer 106 or image server 110, in one embodiment, are analyzed to determine the location of objects in images.
In an alternative embodiment, distance to subject 212 can be used to determine the distance an object depicted in the image is located from camera 102. Distance to subject 212, as described above, can be acquired, in one embodiment, using a distance encoder. In other embodiments, range sensors using acoustic or infrared signals can be used to determine the distance of objects from camera 102. For example, in acoustic range sensors an acoustic emitter emits sound waves which travel to an object at which the camera is pointed. The object reflects the sound waves, some of which travel back toward the camera and are received by the acoustic receiver. The time it takes for the sound waves to travel from the camera to the object and return to the camera is used to calculate the distance of the object from the camera. Infrared range sensors operate in a similar manner. As such, in contrast to focal length 210, distance to subject 212 typically provides an actual distance of an object depicted in an image from camera 102 and the calculations required to determine an object's distance from camera 102 using focal length are not needed.
Although the distance of an object from the camera is determined in step 404, determining the actual location (e.g., geographic coordinates) of the object requires additional information. As such, as shown in
At step 406, a direction from the camera location the object is located is determined based on the camera orientation data. Azimuth 214 indicates a direction in which the camera is pointed when the related image was taken. Azimuth 214, indicates an angle from a particular reference angle. In one embodiment, north is designated zero and angles from north are measured in increasing degrees clockwise from north. For example, zero degrees designates north, ninety degrees designates east, one-hundred eighty degrees designates south, and two-hundred seventy degrees designates west. The accuracy of a particular angle can be designated with a specific granularity. For example, in one embodiment, portions of a degree could be indicated in decimal or using minutes and seconds in addition to degrees.
Determining a location of an object in an image using a location of camera 102, object identification, focal length, and azimuth provides object location information with an accuracy suitable for many uses. However, greater accuracy of a location of an object can be determined by taking into account an elevation angle of camera 102 when an image is taken. For example, a distance between the camera and an object may be determined to be 20 meters. If the image is taken when the camera is horizontal, then the object is 20 meters away from the camera in a determined direction. However, if the camera is tilted up or down, the distance of the object from the camera is a combination of a horizontal displacement and a vertical displacement above or below the camera. If the tilt of the camera (i.e., the elevation angle) and the distance between the object and the camera is known, the vertical and horizontal displacement of the object from the camera can be determined.
In one embodiment, image data table 200 and object data table 700 are stored in image database 112. Information stored in image database 112 can be accessed from computer 106 via network 108 and image server 110. In another embodiment, portions of image data table 200 and object data table 700 may be stored in computer 106. For example, a particular user's images and image data may be stored on computer 106 which allows a user to locally access images and image data. Access to tables 200 and 700 allow a user to search for images based on any of particular entry. For example, a user may search for images containing objects located within a specific distance/radius from a particular location.
It should be noted that, although the methods of
Systems, apparatus, and methods described herein may be implemented using digital circuitry, or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include, or be coupled to, one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.
Systems, apparatus, and methods described herein may be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computers are located remotely from the server computer and interact via a network. The client-server relationship may be defined and controlled by computer programs running on the respective client and server computers.
Systems, apparatus, and methods described herein may be used within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor that is connected to a network communicates with one or more client computers via a network. A client computer may communicate with the server via a network browser application residing and operating on the client computer, for example. A client computer may store data on the server and access the data via the network. A client computer may transmit requests for data, or requests for online services, to the server via the network. The server may perform requested services and provide data to the client computer(s). The server may also transmit data adapted to cause a client computer to perform a specified function, e.g., to perform a calculation, to display specified data on a screen, etc. For example, the server may transmit a request adapted to cause a client computer to perform one or more of the method steps described herein, including one or more of the steps of
Systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable storage device, for execution by a programmable processor; and the method steps described herein, including one or more of the steps of
A high-level block diagram of an exemplary computer that may be used to implement systems, apparatus, and methods described herein is depicted in
Processor 802 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of computer 800. Processor 802 may include one or more central processing units (CPUs), for example. Processor 802, data storage device 812, and/or memory 810 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs).
Data storage device 812 and memory 810 each include a tangible non-transitory computer readable storage medium. Data storage device 812, and memory 810, may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices.
Input/output devices 808 may include peripherals, such as a printer, scanner, display screen, etc. For example, input/output devices 808 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to the user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to computer 800.
Any or all of the systems and apparatus discussed herein, including camera 102, computer 106, image server 110, and database 112, may be implemented using a computer such as computer 800.
One skilled in the art will recognize that an implementation of an actual computer or computer system may have other structures and may contain other components as well, and that
The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention.