Patent Application
20090113350
The present invention relates to providing a visual overview pertaining to hierarchically organized digital objects. In particular, the present invention pertains to the formation and display of ordered visual representations, where, each displayed visual representation symbolizes a different node, wherein all symbolized nodes belong to one level in the object hierarchy. The visual representation can be obtained from nodes descendant from the symbolized node, or generated separately.
Increased computer usage and creation of digital objects along with decreasing costs in removable memory, fixed hard disks, and network access have all contributed to the increase in the number of digital objects any individual person may own or want to access. Computer users have moved from tens or hundreds of files to thousands of files or more. Similarly, digital cameras have decreased in physical carrying size and increased in memory capacity, making it easy for users to carry cameras with them more often and capture more pictures and video clips per picture-taking opportunity. As personal digital media collections increase in magnitude, they are also becoming more and more cumbersome for users to manage and access.
Typically, as in a file system, users can easily view digital objects by browsing a hierarchical directory and file structure. Tree based representations are typically provided for users in a graphical user interface (GUI), making it simple for users to view any objects in any given directory. Unfortunately, it is challenging for users to access object leaf nodes from more than one directory at once. In addition, users are not provided with any visual summary information about what is contained within a directory, especially not for objects that are deeply nested below any given directory. Furthermore, a file system view does not provide flexibility for viewing other hierarchical organizations based on other data facets for the same set of digital objects. That is, if a user creates a folder and file structure based on temporal information such as year, then month, etc., then the user is limited to browsing only by that temporal information within the file browser. They cannot, for example, use the file browser to browse the same files based on a hierarchical content-based categorization of the same data—not without first needing to duplicate files and create appropriate folders. Some modern digital object organization software provide support for viewing different data facets (e.g., for browsing by tags or date in a photo collection), but they do not provide a general approach for browsing any facet of hierarchical data. Accordingly, a need exists in the art for an improved way to review and organize digital objects.
The above-described problems are addressed and a technical solution is achieved in the art by a system and a method for providing a visual summary pertaining to a collection of digital assets, according to the present invention. In one embodiment of the present invention, information pertaining to a collection of digital objects is provided by a computer implemented method including the steps of locating a collection of digital objects that are stored on a computer system and which are organized in a hierarchical fashion, and then simultaneously displaying visual representations of digital objects that belong to a selected level of the hierarchy. This is a useful method because computer users typically group and label text documents and image files in such a hierarchical fashion, wherein nested folders, subfolders, and files are appropriately labeled. Digital objects of the present invention may also include video clips and presentation documents. The visual representations may include thumbnails versions of digital objects, a manually created or computer generated representation, an icon, and a montage.
Further embodiments of the present invention include recognizing depth and height levels of a hierarchy and their interrelationships, and selecting such levels for generating visual representations to depict the levels. Other embodiments of the present invention include recognition of leaf nodes and container nodes of the hierarchy, and their ancestry in the hierarchy. Another embodiment of the present invention is selection of a pattern to display the visual representations, such as in a spiral pattern. The present invention can be used to visually summarize hierarchies based on time, events, people, image and document contents, and geographic location.
Other embodiments of the present invention include a computer system suitably programmed to implement the methods of this invention. Such a computer system includes a processor and memory for organizing and storing a plurality of digital objects in tree hierarchy. The processor executes programs for visually summarizing certain ones of the digital objects that belong to a selected level of the hierarchy. A display screen is coupled to the computer system for displaying visual representations comprising the visual summary. In response to a user selecting a depth of the hierarchy to display, the system can then display a visual summary of the corresponding level of the hierarchy, and so on for additional lower depths. This is a top-down embodiment of the present invention (e.g. hierarchy depth). The system can also start with a visual summary of a lowest level of the hierarchy and successively display; higher levels, which is referred to as a bottom-up embodiment of the present invention (e.g. hierarchy height).
Other embodiments that are contemplated by the present invention include computer readable media and program storage devices tangibly embodying or carrying a program of instructions readable by machine or a processor, for having the machine or computer processor execute instructions or data structures stored thereon. Such computer readable media can be any available media, which can be accessed by a general purpose or special purpose computer. Such computer-readable media can comprise physical computer-readable media such as RAM, ROM, EEPROM, CD-ROM, DVD, or other optical disk storage, magnetic disk storage or other magnetic storage devices, for example. Any other media which can be used to carry or store software programs which can be accessed by a general purpose or special purpose computer are considered within the scope of the present invention.
These, and other aspects and objects of the present invention will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following description, while indicating preferred embodiments of the present invention and numerous specific details thereof, is given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the present invention without departing from the spirit thereof, and the invention includes all such modifications. The figures below are not intended to be drawn to any precise scale with respect to size, angular relationship, or relative position.
The present invention will be more readily understood from the detailed description of exemplary embodiments presented below considered in conjunction with the attached drawings, of which:
It should be understood that the attached figures are for purposes of illustrating the concepts of the invention and may not be drawn to scale.
Embodiments of the present invention relate to an effective method of representing a hierarchically structured collection of digital objects, and for facilitating efficient access to such objects through the selection of individual thumbnails or nodes in the hierarchy. These digital objects are typically media objects such as digital image files, digital video clips, digital audio objects, such as “MP3” files, or other digital documents that can be collected by a user and distributed over a variety of storage media and storage locations. However, these objects need not be limited to multimedia objects. Text files can also be represented according to the methods of the present invention.
When users only have a small number of digital objects, they can easily organize them hierarchically using a standard file and folder representation as typically provided by a modern computer operating system. For example, consider a folder hierarchy for pictures viewed in the File Explorer application of Microsoft Windows XP. Using such a standard file and folder hierarchy, users can access items in a straightforward manner by clicking on individual folders. They can opt to view thumbnails of images or view them in a “filmstrip” format, enabling them to preview images without having to open them. Unfortunately, when users' collections grow to thousands of objects, the simple file and folder access available via the operating system can become cumbersome. In order to access media deep in a file system structure, users have to expand each subfolder along the way. They also cannot view the contents of two subfolders together unless they open two separate File Explorer windows and navigate to the appropriate place. Although modern file system views can provide a visual summary of media files such as showing thumbnails of underlying content on folder icons, such visual summaries are only provided for one level of depth in a hierarchy—i.e., for only the immediate “child” media of a given folder node. Two major limitations of using a current standard file and folder hierarchy organization for accessing hierarchically organized media today are thus: 1) visual representations are not available for deeply nested media; and 2) users are limited to accessing only immediate child media at any given node in the hierarchy (i.e., they do not have easy access to all descendant media from any node at any level in the hierarchy).
In accordance with an embodiment of the present invention, a more compact and efficient method of browsing a hierarchically organized set of documents such as a digital image collection stored in folders and files is to provide a visual summary of nodes of the set, based on user-selected level of the hierarchy to view. The level of hierarchy can be based on node depth, as described in more detail below in relation to
For example, consider the typical case where users organize their photo collection on their home computer in file folders based on date and event. At the top level of their photo collection, they have file folders for each year. Within each year folder, they have folders organized by month and event information (e.g., events such as a birthday party, vacation, wedding, day hike, holiday, etc.). Within each of those folders, they may have additional subfolders, usually depending on the number of pictures captured for a given event. For example, pictures from a vacation to Europe might be in a folder called “May 2007 trip to Europe” and then further divided into country visited folders and country folders could be further divided into major cities, activities or famous landmarks. In accordance with an embodiment of the present invention, the user could choose to view the top level folder nodes which, in this example, would present the user with one representative visual summary object per year. Such a visual summary object could be a single picture from a particular year such as the first picture of the year or a picture from a subfolder containing the most pictures. The visual summary object could also be a visual summary that is created by the system or the user, such as a montage (e.g., a montage composed of pictures from different subfolders). Once presented with the visual summary of years, users can select a year node to view all images captured during that year, independent of how deeply nested those images might be within the year folder.
In the above example, year folders represent nodes at depth=1 in the hierarchy. Moving down one level in the hierarchy (i.e. in all year folders), folders within each year folder (month folders and event information folders) are at depth=2 and can represent user-specified events that took place during that year. Thus, in one embodiment of the present invention, users can choose to view folder nodes at depth=2 to get a visual overview of all major events of all years at once.
The present invention is not limited to browsing the hierarchy from the top-down by depth level, but also provides browsing the hierarchy from the bottom up by node height level. As a brief example, given a sample hierarchy as illustrated in
Output from the computer 10 is typically presented on a video display 52, which may be communicatively connected to the computer 10 via the display interface device 24. The display may also be disposed in the same housing as the computer 10. Internally, the computer 10 contains components such as a CPU 14 communicatively connected to bus 12 and computer-accessible memories, such as read-only memory 16, random access memory 22, and a hard disk drive 20, all communicatively connected to the bus, and which may retain some or all of the digital objects referred to herein. The phrase “computer-accessible memory” is intended to include any computer-accessible data storage device, whether volatile or nonvolatile, electronic, magnetic, optical, or otherwise, including but not limited to, floppy disks, hard disks, compact discs, DVDs, flash memories, ROMs, and RAMs.
The CPU 14 communicates with other devices over data bus 12. The CPU 14 executes software that can be stored on the computer accessible memories. In addition to hard disk drive 20, the computer 10 may also contain other connectible computer-accessible memory drives for reading and writing data and software of various types. This may include a CD-RW drive 30 for reading and writing various CD media 42 as well as a DVD drive 32 for reading and writing to various DVD media 40. Audio can be input to the computer 10 through a microphone 48, or other storage device, for example, an MP3 player, communicatively connected to an audio interface device 26. Audio playback can be heard via a speaker 50, or other listening devices, also communicatively connected to an audio interface device 26. A digital camera 6, and most any of the devices discussed herein, or other image capture or storage device can be communicatively connected to the computer 10 through, for example, the USB interface device 34 to transfer digital objects from the camera 6 to the computer accessible memories and vice-versa. Finally, the computer 10 can be communicatively connected to an external network 60 via a network connection device 18, thus allowing the computer to access digital objects from other computers, devices, or data-storage systems communicatively connected to the network. A “data-storage system” may include one or more computer-accessible memories, and may be a distributed data-storage system including multiple computer-accessible memories communicatively connected via a plurality of computers and/or devices, or over a network. On the other hand, a data storage system need not be a distributed data-storage system and, consequently, may include one or more computer-accessible memories located within a single computer or device.
A collection of digital objects can reside exclusively on the hard disk drive 20, compact disc 42, DVD 40, or on remote data storage devices, such as a networked hard drive accessible via the network 60. A collection of digital objects can also be distributed across any or all of these storage locations.
A collection of digital objects may be represented by a database that uniquely identifies individual digital objects (e.g., such as a digital image file) and their corresponding location(s) in the computer memories. It will be understood that these digital objects can be media objects or non-media objects. Media objects can be digital still images, such as those captured by digital cameras, audio data such as digital music or voice annotations, digital video clips with or without sound. Media objects could also include files produced by graphic or animation software such as those produced by Adobe Photoshop or Adobe Flash. Non-media objects can be text documents such as those produced by word processing software or other office-related documents such as spreadsheets or email. A database of digital objects can be comprised of only one type of object or any combination.
Once a collection of digital objects is associated, such as in a database or by another mechanism of associating data, the objects can be hierarchically organized according to a user created hierarchy (e.g., as in a traditional folder (i.e., directory) and file organization available at the operating system level). Alternatively, a hierarchical organization can be automatically derived based on information about the digital objects. For example, using the creation date of the digital object, a hierarchy of folders or container nodes can be formed based on year, month, and then day of month, only creating container nodes when necessary. That is, when a time-based hierarchy only includes years, months or days for which digital objects exist within the database. For the case of digital image files, the EXIF header, well known in the art, of an image captured from a digital camera can be used to provide the capture date for an image.
In step 305, if the user wants to retrieve only container nodes, then the flowchart continues to step 310. In step 310, if the user is interested in only container nodes at the specified depth=d, then the flowchart continues to step 312 where all container nodes at depth=d are added to NodesList and then the NodesList is returned 335 for display.
In step 310, if the user wants to retrieve the lowest container nodes up to and including the depth=d, then the flowchart proceeds to step 315. The method uses a breadth-first traversal of the hierarchy, by first adding all container nodes at depth=1 to ListToProcess 315, where ListToProcess is the working list of nodes to process. In step 320, the first node of ListToProcess is obtained and the method checks in step 322 to see if the node contains other container nodes and its depth is less than the depth=d specified by the user in 302. If the current container node does not have any children container nodes, then that current node is added to the NodesList in step 326. In step 330, we check to see if there are more nodes in ListToProcess. If no, then all necessary nodes have been processed and we can return the NodeList 335 for display. If yes, then in step 328, the next container node is retrieved from ListToProcess and we continue back through step 322. If in step 322, the current container node does have child container nodes and its depth is less than d, then in step 324 all children container nodes are added to the ListToProcess and the method continues to step 330.
In step 342 of
Once a list of nodes (NodesList) has been identified via one of the flowcharts in
Using the ListToDisplay, the system can present the appropriate nodes from the NodesList and visual summary objects to users.
Example ListToDisplay2 results are provided in
While most of the above examples focus on applying the current invention to data organized in a folder and file hierarchy, the current invention can be applied to any type of hierarchical data. In the case of a photo collection, other sample hierarchical organizations include event-based hierarchy, people-based hierarchy (e.g., based on family trees), content-based hierarchy, time-based and location-based hierarchies. Such hierarchies can be manually created or automatically derived by a suitably programmed computer system. In addition, the same collection of data may have more than one hierarchical organization, thereby allowing a user to browse a collection using the same interface applied to different aspects of the data. For example, assume that
The sample prototype depicted in
A people-based hierarchy could be applied to a collection of digital images by identifying each person in an image and mapping that image to a place in a family tree. A picture with more than one person in it could thus occur as a leaf node in more than one place in the tree. The method described in
A content-based hierarchy could be applied to a collection of digital images based on the content of the images. For example, high-level categories in the hierarchy could include people, nature, animals, man-made objects, art, etc. Our approach could be especially useful for browsing a catalog of clip art where users are not familiar with all of the images in the collection. When users are less familiar with a digital object collection, it can be frustrating if user's keyword searches do not match keywords stored in the system. Browsing can then become tedious as users try to understand the type of taxonomy used to categorize objects. In our approach, users can easily access information at any level in the hierarchy, get visual summary of results and click on any node to retrieve all leaf nodes from that point in the hierarchy.
A time-based hierarchy for a collection of digital images could be automatically derived based on a Gregorian calendar and capture date and time of the digital images, where capture date and time can be extracted from EXIF header information stored in the digital image file as is well known in the art. While calendar-based interfaces to a digital collection are common in modern photo organization software, such interfaces typically focus on one of two mechanisms: A) users specify a date range and all images from that date range are displayed on the screen, or B) users get an overview of their images by year, and they can drill down from year, then to month, then to day to access images from a particular day. For option A, when a large number of images match the given date range, users must scroll through all images. In contrast, our approach only shows one visual summary object per node at a specified depth. Thus, for a date range selected from Jun. 1, 2000 to Sep. 30, 2001, users would see all images captured during that period using many modern photo organization software. Assuming that the user took 800 pictures during that time frame, then all 800 pictures would be displayed. In our approach, if users were looking at the level in the temporal hierarchy associated with month, they would only see at most 16 objects (i.e., one object per month in the period, less if some months contained no pictures). If users wanted more details, they could look at the next level of hierarchy which would include those associated with “week”.
In the case of option B of temporal browsing in typical modern photo organization software, users are typically browsing their collection by year and then month and then day. They cannot easily access a summary of photos taken during the summer, for example. They must go to each month and only view one month at a time. In contrast, our method would allow users to view multiple months at a time at a month summary level, week summary level, or day summary level.
A location-based hierarchy is becoming more commonly used for images and other digital objects that have GPS information associated with them. Such information is used, for example, to plot information on a map of the world, a country map, county map, city map, etc. The current invention could be used with a map display to support location-based browsing. In this context, zooming in on a displayed geographical region is equivalent to filtering the data to the currently displayed geographical region and setting the depth of the browsing used in the flow chart of
Various node details may give users a better overview of, or help them to more efficiently browse, the hierarchically organized digital object collection. For example, when browsing by depth, knowing an individual container node's height provides information on how deep the descendant data of that node goes. Such height information of a node could also indicate the level of detail available for one or more of its descendant objects. When browsing by height, knowing a container's depth indicates how far the current node is from the root node.
The “size” of a container node can be used to refer to the number of descendant digital objects contained by said container node. The size of a container node can be an indicator of importance to a user. For example, in a digital image collection hierarchically organized by date, a really large sized node can indicate an important event such as a wedding, graduation, or a long vacation.
Node details such as the height, depth, or size of a node can be presented to the user through visual characteristics or embellishments to the visual representation of a node. In
Another characteristic that might be of interest to the user is node sibling information. Nodes are considered sibling nodes if they have the same parent. Differently colored lines linking the visual representations could be used to visually indicate sibling information. Alternatively, the system could highlight sibling nodes in response to a user hovering the mouse over a given node.
As data collections grow in size, data filtering becomes more important for improving the efficiency of browsing data. For hierarchically organized data, data filtering has the effect of pruning the tree hierarchy to only include items that match the current filter. Once a hierarchy has been pruned based on data filtering, the methods described in this invention can be directly applied to the pruned hierarchy.
Text-based queries are a common form of data filtering that are also supported in our prototype. For visual media such as images, other more complex data filters could be applied, such as visual search based on a region of interest from a photo, image similarity, face-based search based on a photo of a person, color-based search and so-on.
In
Data filtering mechanisms can be aligned or totally unrelated to a given hierarchical structure. For a temporal hierarchy based on the Gregorian calendar, the bar plot 525 of
It will be understood that, although specific embodiments of the invention have been described herein for purposes of illustration and explained in detail with particular reference to certain preferred embodiments thereof, numerous modifications and all sorts of variations may be made and can be effected within the spirit of the invention and without departing from the scope of the invention. Accordingly, the scope of protection of this invention is limited only by the following claims and their equivalents.
