The present invention relates generally to image capture devices that capture digital images, and particularly to those image capture devices that annotate the captured digital images with data.
In the past decades, digital cameras have replaced conventional cameras that use film. A digital camera senses light using a light-sensitive sensor, and converts that light into digital signals that can be stored in memory. One reason that digital cameras are so popular is that they provide features and functions that film cameras do not. For example, digital cameras are often able to display newly captured image on it's display screen immediately after it the image is captured. This allows a user to preview the captured still image or video. Additionally, digital cameras can take thousands of images and save them to a memory card or memory stick. This permits users to capture images and video and then transfer them to an external device such as the user's personal computer. Digital cameras also allow users to record sound with the video being captured, to edit captured images for re-touching purposes, and to delete undesired images and video to allow the re-use of the memory storage they occupied.
However, the same features that make digital cameras so popular can also cause problems. Particularly, the large storage capacity of digital cameras allows users to take a large number of pictures. Given this capacity, it is difficult for users to locate a single image quickly because searching for a desired image or video requires a person to visually inspect the images.
The present invention provides an image capture device that can analyze a digital image, identify objects in the image, and generate metadata that can be stored with the image. The metadata may be used to annotate the digital image, and as an index to permit users to search for and locate images once they are archived.
In one embodiment, a controller analyzes a captured image to classify one or more objects in the image as being a dynamic object or a static object. Dynamic objects are those that have some mobility, such as people, animals, and cars. Static objects are those objects that have little or no mobility, such as buildings and monuments. Once classified, the controller selects a recognition algorithm to identify the objects.
For dynamic objects, the recognition algorithm may operate to identify a person's face, or to identify a profile or contour of an inanimate object such as a car. For static objects, the recognition algorithm may operate to identify an object based on information received from one or more sensors in the device. The sensors may include a Global Positioning Satellite (GPS) receiver that provides the geographical location of the device when the image is captured, a compass that provides a signal indicating an orientation for the device when the image was captured, and a distance measurement unit to provide a distance between the device and the object when the image was captured. Knowing the geographical location, the direction in which the device was pointed, and the distance to an object of interest when the image was captured could allow the controller to deduce the identity of the object.
Once identified, the device can display the digital image to the user and overlay the metadata on the displayed image. Additionally, the metadata may be associated with the image and saved in memory. This would allow a user who wishes to subsequently locate a particular image to query to a database for the metadata to retrieve the digital image.
The present invention provides a device that analyzes a digitally captured image to identify one or more recognizable objects in the image automatically. Recognizable subjects may include, but are not limited to, buildings or structures, vehicles, people, animals, and natural objects. Metadata identifying the objects may be associated with the captured image, as may metadata indicating a date and time, a shutter speed, a temperature, and range information. The device annotates the captured image with this metadata for display to the user. The device also stores the metadata as keywords with the captured image so that a user may later search on specific keywords to locate a particular image.
The device may be, for example, a digital camera 10 such as the one seen in
Lens assembly 12 usually comprises a single lens or a plurality of lenses, and collects and focuses light onto image sensor 14. Image sensor 14 captures images formed by the light. Image sensor 14 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, or any other image sensor known in the art. Generally, the image sensor 16 forwards the captured light to the image processor 16 for image processing; however, in some embodiments, the image sensor 14 may also forward the light to RF 18 so that it may calculate a range or distance to one or more objects in the captured image. As described later, the controller 20 may save this range information and use it to annotate the captured image.
Image processor 16 processes raw image data captured by image sensor 14 for subsequent storage in memory 22. From there, controller 20 may generate one or more control signals to retrieve the image for output to display 24, and/or to an external device via communication interface 32. The image processor 16 may be any digital signal processor programmed to process the captured image data.
Image processor 16 interfaces with controller 20 and memory 22. The controller 20, which may be a microprocessor, controls the operation of the digital camera 10 based on application programs and data stored in memory 22. In one embodiment of the present invention, for example, controller 20 annotates captured images processed by the image processor 16 with a variety of metadata, and then saves images and the metadata in memory 22. This data functions like keywords to allow a user to subsequently locate a particular image from a large number of images. The control functions may be implemented in a single digital signal microprocessor, or in multiple digital signal microprocessors.
Memory 22 represents the entire hierarchy of memory in the digital camera 10, and may include both random access memory (RAM) and read-only memory (ROM). Computer program instructions and data required for operation are stored in non-volatile memory, such as EPROM, EEPROM, and/or flash memory, while data such as captured images, video, and the metadata used to annotate them are stored in volatile memory.
The display 24 allows the user to view images and video captured by digital camera 10. As with conventional digital cameras 10, the display 24 displays an image or video for a user almost immediately after the user captures the image. This allows the user to preview an image or video and delete it from memory if he or she is not satisfied. According to the present invention, metadata used to annotate captured images may be displayed on display 24 along with the images. The UI 26 facilitates user interaction with the digital camera 10. For example, via the U 26, the user can control the image-capturing functions of the digital camera 10 and selectively pan through multiple captured images and/or videos stored in memory 22. With the UI 26, the user can also select desired images them to be saved, deleted, or output to an external device via the communication interface 32.
As stated above, some digital cameras 10 may come equipped with a variety of sensors such as GPS receiver 28 and compass 30. The GPS receiver 28 enables the digital camera 10 to determine its geographical location based on GPS signals received from a plurality of GPS satellites orbiting the earth. These satellites include, for example, the U.S. Global Positioning System (GPS) or NAVSTAR satellites; however, other systems are also suitable. The GPS receiver 28 is able to determine the location of the digital camera 10 by computing the relative time of arrival of signals transmitted simultaneously from the satellites. In one embodiment of the present invention, the location information calculated by the GPS receiver 28 may be used to annotate a given image, or to identify an object within the captured image.
Compass 30 may be, for example, a small solid-state device designed to determine which direction the lens 12 of the digital camera 10 is facing. Generally, compass 30 comprises a discrete component that employs two or more magnetic field sensors. The sensors detect the Earth's magnetic field and generate a digital or analog signal proportional to the orientation. Upon receipt, the controller 20 uses known trigonometric techniques to interpret the generated signal and determine the direction in which the lens 12 is facing. As described in more detail below, the controller 20 may then use this information to determine the identity of an object within the field of view of the lens 12, or to annotate an image captured by the digital camera 10.
The communication interface 32 may comprise a long-range or short-range interface that enables the digital camera 10 to communicate data and other information with other devices over a variety of different communication networks. For example, the communication interface 32 may provide an interface for communicating over one or more cellular networks such as Wideband Code Division Multiple Access (WCDMA) and Global System for Mobile communications (GSM) networks. Additionally, the communication interface 32 may provide an interface for communicating over wireless local area networks such as WiFi and BLUETOOTH networks. In some embodiments, the communication interface 32 may comprise a jack that allows a user to connect the digital camera 10 to an external device via a cable.
Digital camera 10 may also include a slot or other receptacle that receives a mass storage device 34. The mass storage device 34 may be any device known in the art that is able to store large amounts of data such as captured images and video. Suitable examples of mass storage devices include, but are not limited to, optical disks, memory sticks, and memory cards. Generally, users save the images and/or video captured by the digital camera 10 onto the mass storage device 34, and then remove the mass storage device 34 and connect it to an external device such as a personal computer. This permits users to transfer captured images and video to the external device.
As previously stated, the digital camera 10 captures images and then analyzes the images to identify a variety of objects in the image. Different sensors associated with the digital camera 10, such as GPS 28, compass 30, and DMM 18, may provide the information that is used to identify the objects. The sensor-provided data and the resultant identification data may then be used as metadata to annotate a captured image that identifies the image.
The captured image 40 includes several objects. These are a woman 42, a famous structure 44, and an automobile 46. Image 40 may also contain other objects, however, only these three are discussed herein for clarity and simplicity. When analyzing an image, the present invention classifies the different subjects 42, 44, 46 as being either a “static” object or a “dynamic” object. Static objects are objects that generally remain in the same location over a relatively long period of time. Examples of static objects include, but are not limited to, buildings, structures, landscapes, tourist attractions, and natural wonders. Dynamic objects are objects that have at least some mobility, or that may appear in more than one location. Examples of dynamic objects include, but are not limited to, people, animals, and vehicles.
Based on its classification, the present invention selects an appropriate recognition algorithm to identify the object. The present invention may use any known technique to recognize a given static or dynamic object. However, once recognized, the digital camera 10 may use the information as metadata to annotate the image 40. In
Here, objects 42, 44, and 46 are identified respectively using the woman's name (i.e., Jennifer Smith), the name of the structure in the background (i.e., Sydney Opera House), and the make and model of the vehicle (i.e., Ferrari 599 GTB Fiorano). This metadata, which is displayed to the user, is likely to be remembered by the user. Therefore, the present invention uses this metadata as keywords on which the user may search. For example, the user is likely to remember taking a picture of a Ferrari. To locate the picture, the user would search for the keyword “Ferrari.” The digital camera 10 would search a database for this keyword and, if found, would display the image for the user. If more than one image is located, the digital camera 10 could simply provide a list of images that match the user-supplied keyword. The user may select the desired image from the list for display.
For example, the controller 20 would classify the woman 42 and the vehicle 46 in image 40 as being dynamic objects because these objects have some mobility. The controller may perform this function by initially determining that the woman 42 has human features (e.g., a human profile or contour having arms, legs, facial features, etc.), or by recognizing that the vehicle 46 has the general outline or specific features of a car. The controller 20 would then perform appropriate image recognition techniques on the woman 42 and the vehicle 46, and compare the results to information stored in memory 22. Provided there is a match (box 66), the controller 20 could identify the name of the woman 42 and/or the specific make and model of the vehicle 46, and use this information to annotate the captured image (box 68).
Similarly, the controller 20 would classify the structure 44 in the image as a static object because it has little or no mobility. The controller 20 would then receive data and signals from the sensors in digital camera 10 such as GPS receiver 28, compass 30, and RF 18 (box 70). The controller 20 could use this sensor-provided information to determine location information, or to identify a structure 44 in the captured image (box 72).
By way of example, structure 44 is a well-known building—the Sydney Opera House. In one embodiment, the controller 20 calculates that the camera 10 is located at the geographical coordinates received from the GPS receiver 28. Based on the orientation information (e.g., north, south, east, west) provided by compass 30, the controller 20 could determine that the user is pointing lens 12 in the general direction of the Sydney Opera House. Given a distance (e.g., 300 meters), the controller 20 could identify the structure 44 as the Sydney Opera House. If there are multiple possible matches, the controller 20 could provide the user with a list of possible structures, and the user could select the desired structure. Once identified, the controller 20 could use the name of the structure to annotate the digital image being analyzed (box 74). The controller 20 could then display the captured image along with the window overlay 50 containing the metadata. The controller 20 might also save the image and the metadata in memory 22 so that the user can later search on this metadata to locate the image.
The controller 20 may perform any of a plurality of known recognition techniques to identify an object in an analyzed image. The only limits to recognizing a given dynamic object would be the resolution of the image and the existence of information that might help to identify the object. For example, the controller 20 may need to identify the name of a person in an image, such as woman 42. Generally, the user of the digital camera 10 would identify a person by name whenever the user took the person's picture for the first time by manually entering the person's full name using the UI 26. The controller 20 would isolate and analyze the facial features of that person according to a selected facial recognition algorithm, and store the resultant artifacts in memory 22 along with the person's name. Thereafter, whenever controller 10 needed to identify a person in an image, it would isolate the person's face and perform the selected facial recognition algorithm to obtain artifacts. The controller 20 would then compare the newly obtained artifacts against the artifacts stored in memory 22. If the two match, the controller 22 could identify the person using the name associated with the artifacts. Otherwise, the controller 20 might assume that the person is unknown, prompt the user to enter the person's name, and save the information to memory for use in identifying people in subsequent images.
The metadata used to annotate the digital image is associated with each individual image to facilitate subsequent searches for the image as well as its retrieval. Therefore, the metadata may be stored in a database in local memory 22 along with the filename of the image it is associated with. In some embodiments, however, the metadata is saved according to the Exchangeable Image File (EXIF) data region within the image file itself. This negates the need for additional links to associate the metadata with the image file.
Although the previous embodiments discuss the present invention in the context of a digital camera 10, those skilled in the art should appreciate that the present invention is not so limited. Any camera-equipped device able to capture images and/or video may be configured to perform the present invention. As seen in
In some cases, the digital camera 10, or the cellular telephone 80, might not have the ability to classify and identify objects in an image and use that data to annotate the image. Therefore, in one embodiment, the present invention contemplates that these devices transfer their captured images to an external device where processing may be accomplished. One exemplary system 90 used to facilitate this function is shown in
As seen in
As seen in
In another embodiment, the communication interface 32 in the cellular telephone 80 could comprise a BLUETOOTH transceiver. In such cases, the communication interface 32 in the cellular telephone 80 might be configured to automatically transfer any images or video it captured to a computing device 104 via a wireless transceiver 106. In addition, the user may transfer the captured images and/or video to computing device 104 using the removable mass storage device 34 as previously described. Once received, the computing device 104 would execute software modules designed to analyze the digital image to identify the objects in the digital image. The computing device 104 would then save the metadata with the image and display them both to the user.
The system of
Additionally, the present invention is not limited to annotating still images with metadata. In some embodiments, the present invention also annotates video with metadata as previously described.
The present invention may, of course, be carried out in other ways than those specifically set forth herein without departing from essential characteristics of the invention. The present embodiments are to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.