Patent Application
20090280824
The embodiments relate to identifying the geographic location of events such as photographs or other objects by a mobile wireless device, based on the known locations of wireless access points.
A typical GPS receiver calculates its position using the accurately timed radio signals from four or more GPS satellites. GPS signals are adversely affected by multipath delay of the GPS radio signals when they are reflected off surrounding terrain; buildings, canyon walls, urban streets, hard ground, etc. Moreover, because GPS signals have a very low power level when they reach the Earth's surface, they are almost completely attenuated inside buildings.
An example scenario where this becomes a problem is when photographing museum objects on display in several cities, such as when the photographer is touring on vacation. At the end of the tour, after having taken many photographs, the photographer will be unsure of the location of some of the objects, not having kept a detailed record of every object photographed. Even if the camera had a GPS sensor, that would have been of no avail within the buildings housing the museums, where GPS signals would have been distorted or attenuated.
What is needed is a way to perform geo-tagging of objects created by mobile wireless devices that are either not equipped with GPS sensors or that are used in locations where GPS signals are adversely affected by multipath delay or are attenuated inside buildings.
A method, apparatus, system, and computer program product are disclosed for identifying the geographic location where photographs or other objects are created by a mobile wireless device, based on the known locations of a plurality of wireless access points. In example embodiments, the mobile wireless device includes a digital camera for taking photographs. When the shutter button of the digital camera is actuated and a digital image or object is captured by the camera and stored in a digital image file, the control module of the mobile wireless device determines the current location of the device with respect to a plurality of wireless access points having known geographic locations. In example embodiments, the digital image or object can be post-processed to determine what the location of the device was with respect to the plurality of wireless access points.
In example embodiments, the method can perform the step of determining a current location for the wireless device, for example, by analyzing received signals from each of the plurality of access points, to obtain a distance value from the wireless device to each of the plurality of access points. The method can then calculate a relative position of the wireless device with respect to the plurality of access points. The method can then access absolute or estimated positioning information, such as geographic coordinates, of each of the plurality of wireless access point devices and combine the relative position of the wireless device with the absolute or estimated positioning information of the plurality to obtain an absolute or estimated position of the wireless device, such as its own geographic coordinates.
The relative positioning method may perform a calculation, for example, based on signal strengths, wherein distances are related to the respective signal strengths. The relative positioning method may also perform triangulation based on the direction of the signal. The relative positioning method may also perform triangulation based on both direction and signal strength. Other relative positioning methods can include a proximity algorithm or other deterministic algorithms. The relative positioning method may be by pattern recognition of the received signals, matching the received pattern with stored patterns that are associated with known geographic locations, in a technique known as “fingerprinting”.
The location information can include geographic coordinates and names of places and things near the device's current location. The location information may be stored in the digital image file of the device as embedded metadata along with the stored digital image or other object, geo-tagging the photograph. The geo-tagged photographs may then be displayed on the user's device or wirelessly transmitted by the device. The embodiments are especially useful for geo-tagging photographs taken indoors or nearby tall buildings where GPS signals are distorted or not available.
In embodiments, the location information may be stored in a file separate from, but associated with, the stored digital image or other object in the digital image file of the device and the geo-tagging of the photograph may be performed later. In embodiments, the geo-tagging of the photograph may be performed off-line, when the user uploads the digital images of the photographs and the location information to a personal computer or to a server, for example on the Internet for creating a web album.
In embodiments, the digital image or object and the location information may be stored in a variety of media, for example a random access memory (RAM), a programmable read only memory (PROM), a magnetic recording medium such as a video tape, an optical recording medium such as a writeable CDROM or DVD.
The mobile wireless device 100 includes the digital camera module 105, which includes a lens 68, an electric shutter 69, a CMOS sensor 70, and an analog to digital converter (ADC) 72. The lens 68 converges incident light on the CMOS sensor 70. The electric shutter 69 may be an electromechanical or electro-optical shutter that is opaque to the incident light until actuated by the shutter button 106. The CMOS sensor 70 may be an RGB color filter that converts incident light into electric signals representing red, green, and blue light components. Images are captured by actuating the shutter button 106 to open the electric shutter 69, which exposes the CMOS sensor 70 to incident light refracted through the lens 68. The electric signals representing red, green, and blue light output by the CMOS sensor 70 are converted to digital image or object signals by the analog to digital converter 72 and output to the controller 20. The image sensor 70 may comprise a different type of sensor, such as a charge coupled device (CCD). The digital camera module 105 may be mounted anywhere on the device 100, for example on the front side of the device 100 or connected to the device 100 via a cable or via a Bluetooth or other wireless personal area network (WPAN) link.
The controller 20 can further process the digital image or object signals from the analog to digital converter 72, forming a digital image file by compressing the digital image using the Joint Photographic Experts Group (JPEG) compression algorithm or other compression algorithms and performing other image processing operations on the image file before storing the image file in the RAM 62. The digital camera module 105 may also record motion pictures by periodically capturing a sequence of digital images, for example at thirty images per second, and the controller 20 can further process the sequence as compressed JPEG files or Moving Picture Experts Group (MPEG) files or in another format and store them in the RAM 62.
The mobile wireless device 100 and the wireless access points 150A, 150B, and 150C communicate in a wireless network that can be a wireless personal area network (WPAN) operating, for example, under the Bluetooth or IEEE 802.15 network protocol. The wireless network can be a wireless local area network (WLAN) operating, for example under the IEEE 802.11, Hiperlan, WiMedia Ultra Wide Band (UWB), WiMax, WiFi, or Digital Enhanced Cordless Telecommunications (DECT) network protocol. Or, the wireless network can be a wireless wide area network (WWAN) operating, for example, under a cellular telephone network protocol, for example Global System for Mobile (GSM), General Packet Radio Service (GPRS), Enhanced Data rates for GSM Evolution (EDGE), Code Division Multiple Access (CDMA), Universal Mobile Telecommunications System (UMTS) and CDMA2000. The respective wireless network protocols include provision for communication by the mobile wireless device 100 in the network with the wireless access points 150A, 150B, and 150C by means of the respective protocol data unit (PDU) packets 120A, 120B, and 120C. These examples of wireless network protocols for the device 100 are not meant to be limiting, since it is common for wireless communications protocols to provide for communication between mobile wireless devices and a wired network infrastructure via wireless access points.
Each of these example networks is defined by its respective communications protocol to include the exchange of packets of data and control information between the wireless access point and the mobile wireless devices. Each of the communications protocols defines levels of networking functions and the services performed at each level for the wireless access points and the mobile wireless devices operating using the protocol. Typically, the networking functions include the transmission of packets by the access point having the purpose of announcing its presence to mobile wireless devices within range, either by initiating an inquiry or beacon packet or by responding with a response packet to a probe packet from a mobile device.
The mobile wireless device 100 includes a control module 20, which includes a central processing unit (CPU) 60, a random access memory (RAM) 62, a read only memory (ROM) or programmable read only memory (PROM) 64, and interface circuits 66 to interface with the key pad 104, liquid crystal display (LCD) 102, and the digital camera module 105. The device 100 may optionally include a microphone, speakers, ear pieces, a video camera, or other imaging devices, etc. The RAM 62 and PROM 64 can be removable memory devices such as smart cards, Subscriber Identity Modules (SIMs), Wireless Application Protocol Identity Modules (WIMs), semiconductor memories such as a RAM, ROM, or PROM, flash memory devices, etc. The Medium Access Control (MAC) Layer 14 of the network protocol of the wireless device and/or application program 16 can be embodied as program logic stored in the RAM 62 and/or PROM 64 in the form of sequences of programmed instructions which can be executed in the CPU 60, to carry out the functions of the disclosed embodiments. The program logic can be delivered to the writeable RAM, PROM, flash memory device, etc. 62 of the device 100 from a computer program product or article of manufacture in the form of computer-usable media such as resident memory devices, smart cards or other removable memory devices, or in the form of program logic transmitted over any transmitting medium which transmits such a program. Alternately, the MAC Layer 14 and/or application program 16 can be embodied as integrated circuit logic in the form of programmed logic arrays or custom designed application specific integrated circuits (ASIC). The transceiver 12 in device 100 operates in accordance with the network protocol of the wireless device.
The image management program 30 in device 100 enables the controller 20 to process the digital image signals forming a digital image file by compressing the digital image using the JPEG compression algorithm or other compression algorithms and enables the controller to perform other image processing operations on the image file and stores the image file in the RAM 62. The image management program 30 also enables the controller 20 to process sequences of digital images in a moving picture as compressed JPEG files or MPEG files or another format and stores them in the RAM 62. The image management program 30 can be embodied as program logic stored in the RAM 62 and/or PROM 64 in the form of sequences of programmed instructions which can be executed in the CPU 60, carry out the functions of the disclosed embodiments.
The location program 40 in device 100 enables the controller 20 to determine a current relative position of the device 100 with respect to the plurality of wireless access point devices 150A, 150B, and 150C. The location program 40 can calculate relative positions of the device 100 based on received signal strengths, triangulation based on the direction of the received signal, triangulation based on both direction and signal strength of received signals, and other relative positioning methods such as proximity algorithms or other deterministic algorithms. The location program 40 can calculate relative positions of the device 100 based on pattern recognition of the received signals, matching the received pattern with stored patterns that are associated with known geographic locations, in a technique known as “fingerprinting”. Pattern data is collected on an access points observed at a location and this pattern data is later processed to find a match with a corresponding reference pattern for that location out of many possible reference patterns for locations. The location program 40 can be embodied as program logic stored in the RAM 62 and/or PROM 64 in the form of sequences of programmed instructions which can be executed in the CPU 60, carry out the functions of the disclosed embodiments.
The image transmit program 50 in device 100 enables the controller 20 to upload the geographic location metadata and the associated digital image of a photograph and wirelessly transmit them to other mobile wireless devices or to a wireless access point, for example access point 150A, for applications such as geo-tagging the photograph in a web album server on the Internet. The image transmit program 50 can be embodied as program logic stored in the RAM 62 and/or PROM 64 in the form of sequences of programmed instructions which can be executed in the CPU 60, carry out the functions of the disclosed embodiments.
Different meta data format types can be used for geo tagging. Example embodiments for the positional meta data can include:
Other components that may be included in the mobile wireless device 100 of
The mobile wireless device 100 of
In the example of
Step 206 determines the current location for the mobile wireless device 100 with respect to a plurality of wireless access points 150A, 150B, and 150C. The location program 40 is executed by the CPU 60 in the controller 20 to calculate relative positions of the device 100 with respect to the wireless access points 150A, 150B, and 150C based on received signal strengths (RSS), triangulation based on the direction of the received signal, triangulation based on both direction and signal strength of received signals, or other relative positioning methods such as proximity algorithms or other deterministic algorithms. The location program 40 then accesses absolute or estimated positioning information, such as the geographic coordinates, of each of the plurality of wireless access point devices 150A, 150B, and 150C. The geographic coordinates of each of the plurality of wireless access point devices 150A, 150B, and 150C can be stored in a database accessible by the mobile device 100. The absolute or estimated location information can include geographic coordinates and names of places and things near the device's current location. The location program 40 then combines the relative position of the wireless device 100 with the absolute or estimated positioning information of the plurality of access points 150A, 150B, and 150C to obtain a calculated absolute or estimated position of the wireless device 100, such as its own geographic coordinates.
Alternately, the absolute or estimated location information for the access points can be provided in the packets 120A, 120B, and 120C sent by the wireless access points 150A, 150B, and 150C to the mobile device 100. These packets are buffered in the packet receive buffer 301 in the RAM 62 of
The location program 40 can alternately calculate relative positions of the device 100 with the fingerprinting technique of pattern recognition of the received signals from the access points, matching the received pattern with stored patterns that are associated with known geographic locations.
In Step 208, the location program 40 stores the calculated absolute or estimated position of the device 100 as metadata in the calculated position buffer 302 of the RAM 62 of
In Step 210, the geo-tagged photographs may optionally be displayed on the display 102 of the device 100 and the calculated absolute or estimated position of the device 100 can be optionally displayed along with the photograph. The photo can also be placed on a map based on the location coordinates attached to it.
In Step 212, the image transmit program 50 is executed in the CPU 60 of the controller 20 to upload the calculated absolute or estimated position metadata of the device 100 and the associated digital image of a photograph and wirelessly transmit them to other mobile wireless devices or to a wireless access point, for example access point 150A, for applications such as geo-tagging the photograph in a web album server on the Internet.
In embodiments, the calculated absolute or estimated position of the device 100 or the signal patterns measured by the device 100 may be stored in a file separate from, but associated with, the stored digital image of the photograph, in the digital image file of the device. The geographical location determination and/or geo-tagging of the photograph may be performed later. In embodiments, the geo-tagging of the photograph may be performed off-line, when the user uploads the digital image of the photograph and the calculated absolute or estimated position of the device 100 or the signal patterns measured by the device 100, to a personal computer or to a server 600 on the Internet 602, such as for creating a web album.
In embodiments, the digital image and the location information may be stored in a variety of media, for example a random access memory (RAM), a programmable read only memory (PROM), a magnetic recording medium such as a video tape, an optical recording medium such as a writeable CDROM or DVD.
In example embodiments, the method can perform the step of determining the current location for the wireless device 100 with respect to the plurality of wireless access points 150A, 150B, and 150C, for example, by analyzing received signals from each of the plurality of access points, to obtain a relative distance value of the wireless device 100 from each of the plurality of access points 150A, 150B, and 150C. The location program 40 determines a current relative position of the device 100 with respect to the plurality of wireless access point devices 150A, 150B, and 150C.
In embodiments, the location program 40 can then access absolute or estimated positioning information, such as the geographic coordinates, of each of the plurality of wireless access point devices 150A, 150B, and 150C. The geographic coordinates of each of the plurality of wireless access point devices 150A, 150B, and 150C can be stored in a database in the mobile device 100 or can be provided in the signals sent by the wireless access points 150A, 150B, and 150C to the mobile device 100. For example,
The geographic coordinates of each of the plurality of wireless access point devices 150A, 150B, and 150C can alternately be provided to the device 100 in removable memory devices such as smart cards, SIMs, WIMs, or semiconductor memories such as a RAM, ROM, or PROM.
An example relative positioning method may determine the relative position of the device 100 based on received signal strengths of signals received by device 100 from each of the wireless access point devices 150A, 150B, and 150C, wherein distances are related to the radio frequency power loss between each respective access point and the device 100. The positioning method may also use round trip time and time of flight measurement techniques. The location program 40 can execute a triangulation algorithm to estimate the location of the device 100 to be at the intersection of circles with their centers at each respective wireless access point device 150A, 150B, and 150C, the radius of the respective circle being determined by the respective received signal strength, round trip time, or time of flight measurement.
Another example relative positioning method may determine the relative position of the device 100 based on the time of arrival or time difference of arrival of a reference signal from the device 100 to each of the wireless access point devices 150A, 150B, and 150C. The difference in the time of arrival of the same reference signal at the three wireless access point devices 150A, 150B, and 150C can be used to calculate the relative position of the device 100 with respect to the access point devices. The location program 40 can execute a triangulation algorithm to estimate the location of the device 100 to be at the intersection of circles with their centers at each respective wireless access point device 150A, 150B, and 150C, the radius of the respective circle being determined by the respective times of arrival of the reference signal.
Another example relative positioning method may determine the relative position of the device 100 based on the angle of arrival of a reference signal from the device 100 to each of the wireless access point devices 150A, 150B, and 150C. The difference in the angle of arrival of the reference signal at the three wireless access point devices 150A, 150B, and 150C can be used to calculate the relative position of the device 100 with respect to the access point devices. Another example location measurement technique is measuring the angle of arrival at the user's mobile wireless device 100, if the device 100 is equipped with multiple antennas. The location program 40 can execute a triangulation algorithm to estimate the location of the device 100 to be at the intersection of lines of position with respect to each respective wireless access point device 150A, 150B, and 150C, the line of position being determined by the respective angle of receipt of the reference signal. The positioning method may also perform triangulation based on both angle of arrival and signal strength techniques.
Another example relative positioning method may determine the relative position of the device 100 based on comparing the pattern of signals currently received by device 100 with a map of multiple patterns of received signals previously stored in a database or downloaded from a central server, a technique known as “fingerprinting”. The location program 40 can match the currently received pattern of signals from wireless access point devices 150A, 150B, and 150C with stored patterns in the database, which can be used to calculate the relative position of the device 100 with respect to the access point devices. The stored patterns in the database can also be provided to the device 100 in removable memory devices such as smart cards, SIMs, WIMs, or semiconductor memories such as a RAM, ROM, or PROM. In another example embodiment, the currently measured pattern from a particular location can be uploaded from the user's mobile wireless device 100 to a central server for computation of the position, which can then be downloaded to the user's mobile wireless device 100 and attached to the digital image.
In the example of
In the examples shown in
The wireless access point devices 150A, 150B, and 150C of
After determining the relative position of the wireless device 100, the location program 40 then accesses from each access point the absolute or estimated positioning information, such as geographic coordinates, of each of the plurality of wireless access point devices 150A, 150B, and 150C and combines the calculated relative position of the wireless device 100 with the absolute or estimated positioning information of the plurality 150A, 150B, and 150C to obtain an absolute or estimated position of the wireless device 100, such as its own geographic coordinates.
In
The above discussion has been directed to digital photography. Other example embodiments may use the same method to geo-tag other objects such as short message service (SMS) messages, multimedia messages, or other phone messages. For example, when a recipient receives a phone call or SMS message, he/she immediately sees the geo-tag identifying from where the call or message originates. The SMS or phone message can be sent from the caller's mobile wireless device 100 with the caller's radio environment measurement data attached to it as metadata, and while the SMS or phone message is being routed over the network to the recipient, the caller's geographic location is determined in a network server hosting a location program 40 and related positioning databases. Alternatively, positioning can be determined on the user's or the recipient's mobile wireless device 100. Also, for example, personal notes stored in mobile wireless device 100 can be geo-tagged in the same fashion.
Using the description provided herein, the embodiments may be implemented as a machine, process, or article of manufacture by using standard programming and/or engineering techniques to produce programming software, firmware, hardware or any combination thereof.
Any resulting program(s), having computer-readable program code, may be embodied on one or more computer-usable media such as resident memory devices, smart cards or other removable memory devices, or transmitting devices, thereby making a computer program product or article of manufacture according to the embodiments. As such, the terms “article of manufacture” and “computer program product” as used herein are intended to encompass a computer program that exists permanently or temporarily on any computer-usable medium or in any transmitting medium which transmits such a program.
As indicated above, memory/storage devices include, but are not limited to, disks, optical disks, removable memory devices such as smart cards, SIMs, WIMs, semiconductor memories such as RAM, ROM, PROMS, etc. Transmitting mediums include, but are not limited to, transmissions via wireless communication networks, the Internet, intranets, telephone/modem-based network communication, hard-wired/cabled communication network, satellite communication, and other stationary or mobile network systems/communication links.
Although specific example embodiments have been disclosed, a person skilled in the art will understand that changes can be made to the specific example embodiments without departing from the spirit and scope of the invention. For instance, the features described herein may be employed in networks other than WiMedia networks.
