Patent Application
20070271297
The following description is presented to enable a person of ordinary skill in the art to make and use the inventions. Descriptions of specific devices, techniques, and applications are provided only as examples. Various modifications to the examples described herein will be readily apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other examples and applications without departing from the spirit and scope of the inventions. Thus, the present inventions are not intended to be limited to the examples described herein and shown, but are to be accorded the scope consistent with the claims.
For the sake of convenience, at times, digital photographs or still images are used and described as examples of media objects manipulated by the exemplary system, but those skilled in the art will recognize that the various examples apply similarly or equally to other media objects, subject to appropriate modifications and use of other functions where appropriate (e.g., viewing a media object may apply to viewing a still or moving image or listening to an audio media object, such as a soundtrack).
Clients 110 and media server 100 may communicate, e.g., via suitable communication interfaces via a network 112, such as the Internet. Clients 110 and media server 100 may communicate, in part or in whole, via wireless or hardwired communications, such as Ethernet, IEEE 802.11b wireless, or the like. Additionally, communication between clients 110 and media server 100 may include various servers such as a photograph/image server, map server, mail server, mobile server, and the like.
Media server 100 includes logic or is programmed to process media objects and format data, accessed from local or remote databases or other sources of data, for presentation to users of clients 110, preferably in the format discussed in the examples herein. To that end, media server 100 may utilize various Web data interface techniques such as Common Gateway Interface (CGI) protocol and associated applications (or “scripts”), Java® “servlets”, i.e., Java® applications running on the Web server, or the like to present information and receive input from clients 110. Media server 100, although described herein in the singular, may actually comprise plural computers, devices, backends, and the like, communicating (wired and/or wireless) and cooperating to perform some or all of the functions described herein. An exemplary media server is described in U.S. patent application Ser. No. 11/350,635, entitled “MEDIA OBJECT METADATA ASSOCIATION AND RANKING”, and filed Feb. 8, 2006, which is hereby incorporated by reference in its entirety as if fully set forth herein. Of course, media server 100 may be implemented in various fashions and include various hardware, software, or firmware to carry out the examples described herein.
In one example, media server 100 includes various components known in the art, e.g., a scripting engine (such as a PHP scripting engine), an aggregation engine, and the like. A scripting engine may include authentication logic, upload logic, metadata processing logic, and permissions handling logic. The authentication logic authenticates a user signing on to the media server web site. The upload logic may be used to upload from the client to the server data conforming to any media format, e.g., still image (e.g., JPEG, TIFF), video (e.g., MPEG, AVI), or audio (e.g., MP3, OGG). The aggregation engine may include a statistics engine and a search engine. A client for use with the server of the invention may include a typical web browser application.
Metadata processing logic permits a user to enter metadata to describe each image (as used herein, “metadata” may refer to one metadatum or plural metadata). Metadata may take the form ozone or more tags for a media object. Other types of metadata include, e.g., a title, a description of the image, annotations, and comments. Additionally, metadata associated with a device (e.g., device-supplied metadata) such as metadata from a digital camera that took the picture, including metadata regarding time of capture, aperture, shutter speed, and other settings. Such metadata may be extracted, for example, from the EXIF (Exchangeable Image File) header stored inside the uploaded file by many digital cameras, or the IPTC (International Press Telecommunications Council) header stored inside the uploaded file by many photograph management and organization software packages.
Additionally, metadata may include geographical reference data associated with the media object. For example, the device may generate a geographical location (e.g., from a GPS location device, cell identification, or other location technologies). Additionally or alternatively, a user may enter or edit the geographical location associated with a particular media object (either at the time of capture or later).
One or more clients 110 may transfer media objects and/or related metadata to media server 100 for inclusion within database 106. Further, one or more clients 10 may request summaries of a portion or all of the collection of media objects and/or related metadata included with database 106 or otherwise accessible via media server 100. For example, summarization logic 108 may process a collection of media objects stored locally, remotely, or combinations thereof.
In one example, a system and method are described for selecting and displaying a subset (or summary) of a collection of media objects. In one example, media objects of a collection are geographically referenced and pertain generally to a particular spatial region (e.g., a map or categorization of a specified spatial region). An exemplary method and system select a subset of the collection based on a pattern of the media objects within the spatial region. The pattern may be based on spatial aspects, temporal aspects, social aspects (e.g., groups, contacts, connections and the like associated with a particular user or users), or combinations thereof associated with the media objects and/or user. For example, the summary may be based on a pattern of the media objects, e.g., relating to a density of the media objects within a spatial region at particular locations, and also based on one or more characteristics of a user requesting the summary.
Exemplary summarization logic (included with media server 100, for example) as described in greater detail below, may produce a summary of the media objects as shown in
Additionally, display logic may present or display the media objects as thumbnail images over map 200, and which may be expanded in response to user input (e.g., scrolling over media object 220a generates the display of a larger image 230). It will be recognized that in some example a client may further expand image 230, download image 230 to a user device, transfer (e.g., via email) image 230 to other clients, and the like.
In other examples, the system could summarize a collection of media objects according to a spatial region without displaying a map. For example, a summary of the collection could be categorized within the database or presented under headings according to the summarization processing and spatial region of interest. In yet other examples, the summary of media objects (or at least data associated therewith) may be supplied to a third party, which may present the summary with a map, advertisements, and the like, for example.
An exemplary system and method having summarization logic may be used in a number of applications. For example, such an algorithm may be used for geographic image search, returning a summary of photographs from a region in response to a search query (that can be specified as a text term or a map region), and the like. The algorithm can be used to assist in map-based browsing of images, for example, by enabling a “semantic zoom,” wherein as a user changes zoom levels, panned to any region, the system displays a subset of images that represent that region. For example,
Selecting representative media objects from a collection of media objects and for a given spatial region posses several difficulties. For instance, image analysis alone is generally poor at understanding the semantic content of an image, for example, making purely visual relevance insufficient for summarizing a collection of media objects. In collection from multiple users, the biases of one user's data may also bias the selection towards generally insignificant subjects. The notion of relevance is not well defined, and often subjective; it is therefore difficult for an automated system to learn and assess the relevance of photographs without appropriate models of human interest.
In one example, the summarization logic 108 of media server 100 includes metadata-based heuristics associated with patterns in the photographs and users' photographic behavior. An exemplary heuristic includes the premise that photographs taken at a particular geographical location typically imply the presence of something interesting at the particular geographical location; for example, the attention paid to an object is generally an indicator of its importance. Thus, in one example, the summarization logic includes an algorithm operable to input one or more of a multitude of spatial, social, and temporal metadata, such as where the photograph was taken, by whom, at what time, etc., as well as textual-topical patterns in the data, such as textual tags associated with the photograph (e.g., user inputted tags). Furthermore, the algorithm may be adjusted to bias the set of results using various factors such as the social network distance of the photographers to the user making the query.
It is noted that the specific example and algorithms described are illustrative of a specific implementation of selecting a summary of media objects from a collection of media objects. Those of ordinary skill in the art will recognize that other systems and methods, utilizing various algorithms and inputs related to the media objects or the user, may be used to select summaries based on patterns within the collection of media objects. Accordingly, the following example is not intended to be limiting.
In this example, summarization logic 108 operates on a collection of media objects to produce a subset or summary of the collection of media objects. For instance, given a collection of n photographs, A={P1, . . . , PN}, the exemplary method and system determines a permutation π of A such that any k-length prefix of π(A) is a k-element summary of A. It is desired that a summary, generally referring to a subset of the media collection, capture representativeness, relevance, and breadth in the original collection, for example. In one example, a summary is captured through a subset of one or more metadata attributes that are associated with the photographs. Exemplary metadata attributes may include, but are not limited to, any combination of location (e.g., photograph Pi was taken at location (xi, yi)), time (e.g., photograph Pi was taken at time ti), photographer (e.g., photograph Pi was taken by user ui), tags (e.g., photograph Pi was manually assigned the list of textual tags wi), quality (photograph Pi is associated with an externally derived parameter qi that represents its image quality), bias (also referred to herein as “relevance,” e.g., photograph Pi is associated with a relevance factor ri; relevance may include arbitrary biases (or preferences) based on parameters such as recency, time of day, day of the week, social aspects such as a social network of the user, textual tags associated with that photo, etc.), user profiles, and the like.
The system includes logic for selecting a subset of the collection of photographs following a few heuristics, which may be adjusted by a user or the media server. Exemplary heuristics include, for example, that photographs are taken at locations that provide views of some interesting object or landmark, a location is more relevant if photographs were taken at the location by a large number of distinct photographers, and if available, location-specific patterns of textual tags can reflect the presence of interesting landmarks in a location.
It is noted that a distinction is made herein between the location of the camera (or other device used to capture the media object) and the location of the photographed object. For example, most geographic data associated with photographs pertains to the camera location. In some instances the two locations are near each other (for example, when the photograph is of a house or, say, a fountain), while in other cases (a photograph of, say, the Golden Gate Bridge) the camera may be a considerable distance from the object (e.g., miles away). In one example, the system receives as input the location of the camera, and associates the tags with that location even if the photographed object is remote.
In addition to heuristics listed above, an exemplary system may also operate to represent a broad range of subjects, instead of thoroughly display a few, and allow personal or query bias to modify the system's results. For example, an exemplary system may include the flexibility to allow a user to specify a personal bias, or preference, toward any of the available features of the system and/or to utilize implicit bias in the query context and history. For example, a user may bias by any quantifiable feature of the photographs, such as the social distance of the photographer to the user, preference for weekend photographs, the externally determined quality of the image, and the like.
In one example, the system includes logic for producing an ordering of photographs in the collection. Generally, producing the ordering of photographs comprises two broad processes: clustering the photographs of the collection, and ranking the photographs. These processes may be carried out by clustering logic and ranking logic included with summarization logic 108 (see, e.g.,
In one example, the clustering logic may cluster photographs of a media collection based on their respective geographic locations. This may be accomplished by various known clustering methods. In one example, the clustering logic is operable for clustering the photographs based on their geographical locations into a hierarchical clustered structure (for example, having a series of subclusters). The exemplary system further includes ranking logic for ordering the photographs within the cluster generated by the clustering logic. An ordering of the photographs may be generated, for example, by interleaving photographs from the collection subclusters, prioritized according to a calculated score. The ranking logic may then recursively rank the photographs for each of the subclusters, starting from the leaf clusters. A summarization of the photographs, e.g., via a display of the photographs, for a specified spatial region may then be generated based on the ranking of photographs within the collection subclusters.
To illustrate the summarization process and scoring mechanism, an exemplary hierarchical cluster structure is illustrated in
In one example, the clustering logic and particular clustering algorithm may include a Hungarian method, which is known in the art, or a modification thereof. One particular modification to the Hungarian method includes the method described in “The Hungarian Clustering Method,” by Jacob Goldberger and Tamir Tassa, the content of which is published at, http://www.openu.ac.il/Personal_sites/tamirtassa/Publications/hcm.pdf, and which is incorporated herein by reference. Other exemplary clustering methods may include conglomerative clustering, partitional clustering, hierarchical agglomerative, divisive clustering, or the like.
In this example, the clustering logic is generally operable to identify a hierarchy of clusters within a given dataset (e.g., generated from the collection of media objects) of points in n dimensions, based only on the distances between those points. The input to that algorithm is a set of points in the plane
A={(xi,yi)ε ,1≦i≦n} (1)
and the output is a clustering of those points C(A), where C(A) is a tree; each node in the tree represents a subset of A; and the children of each node are a partition (or clustering) of the subset that is associated with that node.
Given a hierarchical clustering based on the locations of the photographs, for example, the ranking logic is operable to produce a ranking of each object or item in the collection. In one example, the ranking logic first assigns a score to each cluster, and performs a recursive interleaving algorithm that uses the clustered structure and the corresponding scores in order to produce a natural flat ordering. For example, given a hierarchical clustering C(A) of A, where A is possibly itself a subcluster in a larger hierarchy, the algorithm outputs a permutation π on A that represents ranking on A. That ordered sequence is denoted henceforth by S=π(A).
Assume that m subclusters are identified in A, A=Ui−1m=Ai, and that
f(A1)≧f(A2)≧ . . . f(Am)
where f represents a black box determining the cluster score. In one example, each cluster represents a different site or focal point in the area where the photographs were taken. Moreover, the subclusters of each of the clusters may represent different locations within the corresponding site.
In one example, it is desired that the ranking logic produce a ranking that would balance the contradicting properties of depth and breadth of coverage. For example, consider the second level of the exemplary hierarchy shown in
Accordingly, in an attempt to balance some amount of depth, breadth, and consistency according to one example, the summarization logic and algorithm assumes that the photographs in each subcluster of a current A have been ranked recursively according to the algorithm. The ordered sequence of photographs for A will then have two parts: a short header H followed by a trailer T, where S(A)=H∥T.
The header H will include a photograph from all prominent clusters. To that end, a threshold 0<w<1 is defined and a cluster Ai is deemed prominent if
Assume that there are m′ prominent clusters among the m clusters, with 0≦m′≦m. Then the header is
H=(A1,1,A2,1 . . . Am′,1),
where Ai,j means the jth most relevant photograph from cluster Ai.
Next, a trailer is generated. In one example, each cluster is assigned a probability that equals its score divided by the sum of scores of all clusters. These probabilities are then used to randomly select a cluster. If cluster Ai was selected, its top-ranked remaining photograph is removed and appended to T and the score and probability of that cluster is recalculated to reflect the removal of its top photo. The process of selecting photographs and appending them to the trailer is repeated until all photographs have been selected.
The score of a cluster Ai may depend on several factors or attributes of the clusters, including, for example, the following (which will be described in greater detail below):
The sum of bias factors (e.g., relevance) of all photographs in the cluster,
The tag-distinguishability of the cluster, τi.
The photographer-distinguishability of the cluster, øi.
The density of the cluster. More specifically, let σx,i and σy,i denote the standard deviation of the x and y coordinates, respectively, of all points in Ai, and let
σi=((σx,i)2+(σy,i)2)1/2.
The cluster density may be defined as
δi=1/(1+σi). (2)
The sum of image qualities of all photographs in the cluster,
It is noted that while most of the above factors may be derived from data that is contained in the collection of media objects (e.g., included with the metadata), the bias factor depends on subjective requirements that are introduced through the query to the photograph collection (e.g., set by the system for a particular application or by a user via their particular request).
Bias (relevance) Factors: In one example, the bias factor ri of a photograph Pi may depend on one or more parameters such as recency, the time of day, the time of the week, the photographer, textual tags associated with that photo, and the like. Each photograph may be assigned a score θ(Pi) in the range [0,1] for each such parameter. The final bias score can be a weighted average of all these scores, which is the bias score ri of photograph Pi. The amount to which each feature contributes to the overall bias calculation can be specified by a user, or set by the system according to the application.
For example, a user may request photographs taken during the day, or by a set of one or more photographers (e.g., their contacts, friends, family, etc.). These inputs will weight the bias factors accordingly, thereby biasing the results per the user query.
Tag distinguishability of the cluster: In one example, tag distinguishability aims to detect distinct or unique concepts that are represented by a given cluster by examination of the tags. The assumption is that the more unique concepts a cluster represents, the more semantically meaningful a cluster tends to be. For example, in
Formally, each photography Pj, 1≦j≦n, is tagged with tags that are drawn from a finite dictionary. Let Ft,i denote the frequency (count) of t in Ai, (the usual term frequency of the tag/term t in the cluster/document Ai, as often used in information retrieval). There are several ways to measure how the term t distinguishes the cluster Ai from other clusters. Denoting such measures by Tt,I; the usual measure in information retrieval is the tf-idf weight (where term frequency—inverse document frequency is also familiar and well known in the art of information retrieval). The measure is defined as
T
t,i
:=tfidf
t,i
=tf
t,i
·idf
t, (3)
where, tft,i32 Ft,i, while
is a measure of the overall frequency of the tag t in the entire photograph collection. Another alternative to (3) which is used in information retrieval is
T
t,i
:=tfidf
t,i
=tf
t,i·log(idft). (5)
Large values of both weights indicate that the number of occurrences of t in Ai is large with respect to its number of occurrences elsewhere.
It is noted that in the usual tf-idf weight, the document-frequency is defined as the frequency of clusters in which the tag appears, as opposed to the frequency of actual tag occurrences. However, the usual definition is generally not suitable for cases where the number of clusters (documents) is small. In such cases, a single random occurrence of a tag in a cluster may have a significant effect on the usual measure, while in this instance it would be hardly noticeable.
Next, an overall tag-distinguishability measure for Ai, Ti, is defined based on the tag-distinguishability measures of all tags in the cluster, {Tt,i: t ε T}. The overall score may be computed by using the Euclidean measure based on the 12-norm,
Photographer distinguishability of clusters: In one example, a measure of photographer distinguishability (or user distinguishability) is determined, which may indicate how much a given cluster is associated with a small number of photographers, making the cluster less likely to be semantically meaningful. For example, in
Hence, much like for tags, a tf-idf-like score for the correlation between a cluster Ai and a photographer u may be used. For example, let Bu denote the set of locations of photographs that were taken by the photographer u (as with the set A, (1), of all photograph locations, Bu is a multiset, in the sense that it may include repeated values). The score may then be given by
φu,i:=tfu,i·idfu (7)
where
and
Note that (7) and (9) are equivalent to (3) and (4), respectively. As previously, compare (5) with (3), and (7) may be replaced with
φu,i:=tfu,i·log(idfu). (10)
Finally, the overall photographer-distinguishability may then be defined, in this example, as:
When dealing with tag-distinguishability, the more a given cluster is associated with some tags, the more it is conceived as a ‘well-defined’ cluster that has prominent distinguishability from other clusters. Photographer-distinguishability, on the other hand, has an opposite effect. The more a given cluster is associated with a single photographer (or few photographers), the less interesting that cluster becomes.
Overall Cluster Score: The score s(Ai) of the cluster Ai depends, in one example, in a monotonically increasing manner on the bias factor, ρi, and the image quality factor, κi. It also depends in a monotonically increasing manner on the density measure of the cluster, δi, (2). The score s(Ai) may also depend in a monotonically increasing manner on Ti, the tag-distinguishability measure of the cluster. Finally, the score may depend in a monotonically increasingly manner on øi−1, the inverse photographer-distinguishability measure of the cluster, as discussed above. Therefore, the overall score may be represented as
h(κi,δi,Ti,φi−1)·ρi (12)
where h is, for example, a weighted average of its variables, and the weights may be chosen and fine-tuned by experimentation.
It will be understood that the above scoring methods are illustrative of some exemplary methods in which a system may score clusters, and is not intended to be exhaustive or limiting. In other examples, additional or different factors may be used. Additionally, similar factors as described herein, such as bias factors, tag-distinguishability of clusters, etc., may be determined in alternative manners.
A media server including summarization logic as described herein has a number of possible applications and uses. In one example, the media server and summarization logic may be used to as part of or to support a “semantic zoom” application using a large collections of digital images (or other media objects), or assist in browsing/searching a large collection of media objects by initially showing summarized results.
Semantic zoom generally refers to changing the space of interest in which the media objects are related, such as a map, timeline, or combination thereof, where the zoom or change in space is accompanied by a corresponding shift in the level of content represented on the map or timeline. In the example of a spatial map, this may include presenting to a user a number of photographs appropriate to an initial or current zoom level for a given spatial map. The system selects images for display according to the summarization logic (e.g., based on a determined pattern, the particular ranking, etc.). As the user zooms in, more photographs (e.g., that were ranked lower) are revealed, thus bringing the content into more detail. At any zoom level, panned to any region, the system and associated logic automatically displays a summary of photographs that represent that region per the particular summarization logic inputs. Given an ordering on a collection of photographs, a system may quickly display a summary of photographs for varying levels in response to user input. When viewing any region, the system may display the k best photographs that were taken within that region. Alternatively, the system may display a fixed number of photographs for any zoom level, letting the number of photographs that appear in a particular region at that level implicitly correspond to the relevance of that particular region.
In addition, the exemplary summarization logic and algorithm described may be used to generate or seed “tag maps,” e.g., a visualization for geo-referenced collections based on textual tags that are associated with the data.
In this example, exemplary summarization logic and methods as described herein generated tag map 400, which represents photo-taking in San Francisco, Calif. Rather than display representative photographs at their respective locations (e.g., as shown in
In one example, in the tag-distinguishability process described previously, a tag-distinguishability score τi is calculated for each cluster Ai at each level of the hierarchy. In the process, an individual score τt,i is calculated for each tag t. This tag score, as a variant of tf-idf, can be thought of as a measure for how well and how uniquely t represents Ai. Thus, consider the following mapping: for some ‘natural’ level of the hierarchy, (to be determined later) let qi be the tag t ε T that maximizes τt,i. The displayed size of the tag corresponds to its score. In this example, the tags are displayed above the centroids of their given cluster on the map. Additionally, in other examples, other types of information can be encoded in the tag map; for example, recency can be represented via the color of the tag text.
Various aspects described may also be combined in various manners. For example, a tag map can be displayed or selected at different zoom levels via process and methods describe with respect to the zoom aspects. Moreover, tag maps may be used for collections other than geo-referenced photographs: e.g. for visualizing popular search keywords from different areas on the map.
Additionally, the various aspects and examples may be combined or used with other known methods and system. In one example, the exemplary system may operate to select a subset of a collection of media objects (stored remotely or locally) as described herein in response to a search request or function. For example, a subset may be returned or included with search results when a text, image, or other search is performed on an object that is recognized by the system as a spatial object (e.g., Germany, San Francisco, Central Park, or the like).
The various aspects and examples of the inventions can be implemented in any suitable form including hardware, software, firmware or any combination of these. Different aspects of the invention may be implemented at least partly as computer software or firmware running on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally, and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units and processors.
Although the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the claims. Additionally, although a feature may appear to be described in connection with a particular embodiment, one skilled in the art would recognize that various features of the described embodiments may be combined in accordance with the invention. Moreover, aspects of the invention describe in connection with an embodiment may stand alone as an invention.
Moreover, it will be appreciated that various modifications and alterations may be made by those skilled in the art without departing from the spirit and scope of the invention. The invention is not to be limited by the foregoing illustrative details, but is to be defined according to the claims.
