Patent Application
20110285748
Most people have large collections of poorly indexed personal media, much of which has high emotional appeal but relatively low visual quality. Skilled practitioners can produce visually appealing artifacts (e.g., composite images and slideshows) from such collections but few users have the time or creative ability to do so themselves. Conventional approaches to screen-based photo display tend to fall into two categories: manually created slideshows, showing a single image at a time with some sort of transition between them; and automatically generated videos of image sequences, possibly overlaid with some visual effects and set to music. Random photo selection screensavers fall somewhere between these categories.
What are needed are improved systems and methods for displaying images.
In one aspect, the invention features a method in accordance with which a sequence of templates is determined. Each of the templates defines a respective spatiotemporal layout of regions in a display area. Each of the regions is associated with a respective set of one or more image selection criteria and a respective set of one or more temporal behavior attribute values. For each of the templates, a respective image layer for each of the regions of the template is ascertained. In this process, for each of the layers, a respective image element is assigned to the respective region in accordance with the respective set of image selection criteria. For each frame in a sequence of frames, a respective set of rendering parameter values is produced. Each set of parameter values specifies a composition of the respective frame in the display area based on a respective set of ones of the image layers determined for one or more of the templates and the respective sets of temporal behavior attribute values. The sets of rendering parameter values are output to a composite image renderer.
The invention also features apparatus operable to implement the inventive method described above and computer-readable media storing computer-readable instructions causing a computer to implement the inventive method described above.
Other features and advantages of the invention will become apparent from the following description, including the drawings and the claims.
In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.
A “template” is a content-free specification of a spatial layout of content regions each of which may be associated with respective temporal behavior and other attributes.
As used herein, an “image element” is a reference to a portion of an image and may be associated with one or more attribute values.
A “frame” is a complete image that is composited from one or more layers.
A “layer” is an object that defines different elements of a composite image.
A layer can have a certain transparency/opacity and a number of other properties, including spatiotemporal properties.
A “matte” is linked to a layer and defines a mask that indicates the contribution (e.g., level of transparency) of pixels of the layer to a final composite image.
The term “z-order” means the order in which layers are combined to form a composite image.
A “non-photorealistic” variant of an image refers to a version of the image that purposefully and stylistically modifies the image.
As used herein, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on.
Most people have large collections of poorly indexed personal media (particularly digital photographs and digital video clips), much of which has high emotional appeal but relatively low visual quality. The increasing availability of large electronic display spaces in the home supports the delivery of ambient experiences based on these media collections. The embodiments that are described herein are operable to automatically generate a dynamic collage of images (e.g., image elements, such as digital photographs and salient elements extracted therefrom). The collage can be configured to change continuously over time so as to produce a visually appealing ambient experience that is likely to sustain a viewer's interest over long time periods. These embodiments typically have imperfect knowledge of what the user is likely to find interesting. However, by extracting multiple objects of interest from user photos and displaying them together, these embodiments typically are able to provide the user with a rich choice of visual targets such that the user is more likely to see at least one thing of interest in the dynamic collage. In some cases, a dynamic collage specification can leverage the large number of possible variations in the presentation of visual content to reduce the likelihood that the same composition will be presented more than once, thereby ensuring that the presentation of an image collection always has a refreshing appeal.
Some embodiments include an adaptive background generator that generates multiple background elements that can be visually integrated with the presentation of the image elements so that the overall impact of the display is pleasing. Backgrounds typically are automatically designed to suit the collage arrangement, avoiding the overlap of high interest areas in background and foreground.
In some embodiments, designer templates can be used to specify image selection criteria and temporal behavior attributes for producing a themed collage offering high quality images for print. After sufficient time has been allowed for the user to take in a collage composition produced in accordance with one template, user interest is sustained through a content-driven transition taking the viewer into another composition that is produced in accordance with the same or a different template. These embodiments are able to automatically continue and develop a specified theme (e.g., faces from photos of the same person) from one collage to the next.
In accordance with the method of
For each of the templates 22, the collage layer generator 12 ascertains a respective image layer for each of the regions of the template (
For each frame in a sequence of frames, the collage layer generator 12 produces a respective set of rendering parameter values (
The collage layer generator 12 outputs the sets of rendering parameter values to the composite image renderer 14 (
As shown in
As explained in detail below, each of one or more of the image elements 30 may be associated with a respective set of mattes that demarcate the image element in a respective source image, and a respective set of one or more non-photorealistic variants of the source image. The mattes typically are non-rectangular; although in some embodiments, the mattes are rectangular. The image and matte elements are passed to the composite image renderer 14 as a sequence of layers which overlay each other according to a defined z-order. Each layer has timing specifications determining how the elements move, scale, blend, and how the alpha matte is applied. By programming the timing of adjustments to these parameters the collage layer generator 12 can generate interesting transitions from one island to another, and create continuous cyclic motion within each island of stability.
In addition to image-based layers, some embodiments of the collage layer generator 12 create artistically inspired backgrounds that are designed to accommodate the dynamic locations of the image elements. Backgrounds typically are composed of multiple layers, each with its own z-position, motion, matte, and timing data values. This permits effects such as background details which move continuously, allowing overlaying and blending with the image layers.
A. Introduction
The source image processor 44 generates the image database 32 from source images 46, which typically includes the user's source material. The source images 46 may correspond to any type of digital image, including an original image (e.g., a video frame, a still image, or a scanned image) that was captured by an image sensor (e.g., a digital video camera, a digital still image camera, or an optical scanner) or a processed (e.g., sub-sampled, filtered, reformatted, scene-balanced or otherwise enhanced or modified) version of such an original image. In the process of building the image database 32, the image element extractor 44 extracts metadata from the source images 46. Among the exemplary types of metadata that are extracted by the source image processor 44 are face locations, face identity, facial feature points, temporal data (e.g., acquisitions times), non-facial saliency data, segmentation data, color analysis data, and facial expression information. In some embodiments, the source image processor 44 may extract metadata from sub-sampled versions of the source images 46.
The collage layer generator 42 includes a scheduler 48, a template filler 50, a matte generator 52, a non-photorealistic variant generator 54, and a background generator 56. The scheduler 48 creates a sequence of islands of stability. As explained above, each island is a collage of a set of image elements 30 that are arranged according to a respective one of the designer templates 22. The image elements are selected by the template filler 50 according to a theme that gives semantic coherence to each collage. Examples of themes include photos of the same person and photos of a person with their friends. The matte generator 52 generates a respective set of non-rectangular mattes that reference a respective one of the source images 46 and defines a respective one of the image elements 30. The non-photorealistic variant generator 54 generates a respective non-photorealistic variant of an original source image and associates the non-photorealistic variant with the corresponding image elements. The mattes and the non-photorealistic variant typically are passed to the composite image renderer 14 as a sequence of layers that overlay each other according to a defined z-order. Each layer has timing specifications determining how the elements move, scale, blend, and how the alpha matte is applied. By programming the timing of adjustments to these parameters the scheduler generates interesting transitions from one island to another, and creates continuous cyclic motion within each island of stability. These transitions may make use of the meta-data in the image database 32; for example, zooming into a particular face from an image element in the current collage, aligning the eyes to a those of a face from a different image element in the new collage, blending from one face to the other, and then moving the new face into position in the new collage.
The background generator 56 creates artistically inspired backgrounds that are designed to accommodate the dynamic locations of the image elements. Backgrounds typically are composed of multiple layers, each with its own z-position, motion, matte, and timing data values. This permits effects such as background details which move continuously, allowing overlaying and blending with the image layers. In some embodiments, the background generator 12 uses an avoidance map for the current island of stability to ensure that important areas of the image elements (e.g., faces) are not obscured during the presentation of the dynamic image collage.
The composite image renderer 14 takes the layer specification provided by the scheduler 48 and produces the frames that are rendered on the display 16 in real time. In some embodiments, the composite image renderer 14 is implemented by a commercially available rendering system (e.g., a video or graphics card that is interfaced via an API, such as the DirectX® API available in Microsoft Windows® operating systems) that support layer combination processes and allow the properties of each layer to be specified as a function of time. These properties include size, rotation, displacement, z order, and transparency. In these embodiments, the graphics capability of the rendering system can achieve realtime motion, scaling, blending and matting of multiple image layers at high rendering rates (e.g., thirty frames per second or greater).
In some embodiments, the composite image renderer 14 is responsive to two forms of user interaction. The first form of interaction involves the use of a user interface that provides a small set of user commands that enable the dynamic image collage that is being displayed on the display 16 to be frozen at any point in time. The user interface allows the viewer to step backwards and forwards through the sequence of frames to locate a specific frame, in order to save or print it, (potentially re-rendered at higher resolution). The second form of interaction involves the use of a camera and a realtime face detector that feeds back the size of the largest face in the audience. The Renderer can react to this by; for example: a) slowing down the rendering rate as a face approaches the screen, allowing viewers to pay more attention to the frame that caught their attention: b) increasing the size of a previously designated “primary” photo on the display; and c) causing the primary photo to blend from the manipulated version to the original. Additional details about the construction and operation of these embodiments can be found in U.S. application Ser. No. ______, filed on even date herewith, and entitled “SYSTEM AND METHOD FOR DYNAMIC GENERATION OF MULTIMEDIA CONTENT” [Attorney Docket No. 200802868-1].
B. Spatiotemporal Templates
The spatiotemporal templates 22 encapsulate automated design knowledge that controls many aspects of the dynamic image collage generation process, including collage layout, image element selection, choice of non-photorealistic variant generation, and background design.
A template may be specified in any number of ways. The Appendix attached hereto contains an exemplary XML (eXtended Markup Language) template specification. The exemplary template specification defines a set of regions, each of which contains one or more image elements. Each region definition includes constraints on the number of faces allowed, the range of face sizes allowed, and potentially the allowable poses of those faces (ipr=“in plane rotation”). For example, the idmatch element indicates that for the specified region, there are constraints on the identities allowed in the other regions. A “match” child indicates that the match subregion must contain an individual represented in the idmatch region. A “nomatch” child indicates that no individual in the match child region can match any individual in the idmatch region. The “left” parameter specifies the leftmost boundary of the width×height bounding box for the region. The “top” parameter specifies the topmost boundary of the width×height bounding box for the region. The “width” specifies the maximum allowable width for a scaled image element suitable for the region. The “height” specifies the maximum allowable height for a scaled image element suitable for the region. The “count” parameter specifies the number of faces allowed in the region. The “radius” parameter specifies the range of allowable face radii allowed in this region. The “ipr” parameter specifies the range of allowable pose angles. Some embodiments additionally include a “hoop” (=“horizontal out of plane” rotation) parameter that specifies the range of profile to portrait poses allowable in the region.
The “configs” element at the end of the exemplary template specification contained in the Appendix includes one or more config elements that are made up of a subset of the regions that are defined in the template. The config elements specify the regions that are activated in that particular configuration, and enforce cross-region requirements.
Referring to 2, Transition 2
3, . . . , Transition N−1
N) that link the templates in the sequence 60. The sequence of templates can be specified by an experience designer based on the type of experience the designer wishes to create (e.g., a story to be told). Alternatively, the selection of templates may be driven at least in part by an automatic analysis of the source images 46.
As explained above, the dynamic collage experience consists of a sequence of groups of frames. In a typical implementation, content frame groups are separated by transition frame groups, where each frame in a content frame group is rendered by combining image layers belonging to that content group, and each frame in a transition frame group is rendered by combining layers from the adjacent content frame groups. Content frame groups can include image element layers and ornamental layers.
Image element layers in a content frame group are defined by a respective one of the templates 22 that specifies the spatiotemporal layout of regions: each of the regions is associated with image element selection criteria and a respective set of one or more temporal behavior attribute values: and the image elements are assigned to the regions in accordance with the respective sets of image element selection criteria. Each region typically is allocated to its own layer to allow independent motion and combination. Where the image element is a portion of a larger source image, the whole source image typically is loaded into the layer and a mask is associated with the layer to reveal only the required image element. This allows the rendering engine to make transitions between showing only an image element and showing its entire source image by changing a parameter which determines how opaquely the mask is applied.
Ornamental layers have graphics/image objects assigned to them according to a style template and the image elements that have been assigned to the foreground layers. The color of ornamental objects can be chosen to complement the image element layers and the decorative objects can be positioned to avoid visual clashes with image elements, typically based on avoidance maps.
Transition frame groups are rendered from the layers of adjacent content frame groups according to a transition plan. In some embodiments, the layers of both content frame groups will be active and the plan specifies motion and transparency attributes of the old layers as they fade away and of the new layers as they fade in.
In some embodiments, the sequence 60 of templates and transitions is generated at random. The transition is selected first, as this determines whether the next island uses the same template or a different template, (and whether it uses the same or different image elements). Transitions may be selected randomly from a set of possible transitions in a weighted random manner. In one exemplary embodiment, the three possible transitions are:
In some embodiments, a process or script controls the template selection process. In one such embodiment, a process analyzes the source image collection to determine a series of “stories” consisting of sequences of image element sets. Each set of image elements is assigned to a respective one of the templates 22 based on the contents of the images. In some cases, the template is created to suit the selected image elements.
C. Matte Generator
As explained above, an image element typically is a portion (or fragment) of a source image that captures some salient (or interesting) feature of the source image. Examples of salient features include groupings of individuals (preferably cropped out in a visually appealing way), salient foreground objects, or even background scenes. In the illustrated embodiments, the image elements are defined with general crop boundaries, which may be non-rectangular and not even piece-wise linear. In this way, the image elements are interesting to look at and encapsulate the subject in the most visually appealing way. Indeed, the use of image elements with their natural border (e.g. the natural shapes of faces) adds interest beyond the usual rectangular combination of intact source images.
In some embodiments, image elements are generated automatically by the source image processor 44 (
In the illustrated embodiments, image elements are extracted based on the location and size of faces in the source images 46. In this process, the boundary of an image element is determined from a set of face locations and rules that determine how the face locations are combined (or clustered). In some embodiments, these rules require that:
The first condition prohibits half-faces from appearing in image elements, or unnaturally tight crops around faces. The second condition prohibits face based image elements with one face separated from a group. The third condition prohibits image elements with unnatural face crops due to the image border. In other embodiments, different or other conditions could be imposed depending on the requirements of the application.
In these embodiments, the process of allocating faces to image elements involves clustering face locations based on a “distance” between a face and an image element. Exemplary distance metrics include one or more of: the distance between the face and the nearest face in the image element; the distance between the face and the furthest face in the image element; and the distance from the face to the nearest image border. In some of these embodiments, the clustering process is performed in a greedy fashion by comparing all the distance pairs. Note that different image elements can overlap, and they may include the same faces.
In accordance with the embodiment of
The matte generator 52 removes any faces that are too close to edges of the source image (
The matte generator 52 creates image elements around the remaining faces in the source image (
The matte generator 52, then attempts to merge each image element with every other image element in the pool. In this process, each image, element is compared with every other image element to see if the image element bounding boxes overlap more than 10% (
Once a set of locations has been collected, the image element boundary is created. This may require more metadata from the image, and about the specific locations involved. For example, a good image element boundary should be within a specified distance tolerance away from each of the faces in the image (not too close or too far), it should mimic the natural curves of the image element subject (the faces) subject to some smoothness constraints, and it should try to pass through uninteresting parts of the image such as dark areas or flat background. These constraints can be met by using the face data (including things like face orientation so shoulders can be admitted if required) together with a texture map (and possibly some color information) to determine the “flat” areas of the image.
In some cases the constraints supplied may change the locations allocated to the image element. For example, if a very tight crop around faces is required, then it may be that only faces very close together will be admitted to an image element.
After the image element boundaries have been determined, the matte generator 52 associates a respective avoidance mask piece with each image element. Each avoidance mask piece shows the parts of the image element that will be visible, and how important it is for background elements to avoid them. In some embodiments, the avoidance importance is indicated by labeling areas of the image element with one of three gray values: “white”, which means that the labeled area can never be obscured by ornamental elements; “gray”, which means that the labeled area may be obscured by some ornamental elements; and “black”, which means that the labeled area can be obscured by any ornamental elements.
D. Non-Photorealistic Variant Generator
The non-photorealistic variant generator 54 may generate a non-photorealistic variant of source image using any of a variety of different non-photorealistic rendering techniques, including image processing techniques (e.g., stippling, watercoloring, stylizing, and mosaicizing) that transform an image into a style that suggests, for example, a painting or other artistic styles.
E. Background Generator
The background generator 56 is responsible for creating the non image element (or ornamental) layers of an island of stability. The background generator 56 typically generates the ornamental layers so that they do not obscure salient parts of the Image elements.
In some embodiments, the background generator 56 avoids obscuring image elements with ornamental features by ensuring that image element layers are above the ornamental layers in the z-order.
In other embodiments, the background generator 56 interleaves the image element layers and the ornamental layers to form a more pleasing, integrated composition. In these embodiments, some background elements can be positioned above the image elements in the z-order, but they are positioned in such a way as to not obscure the most significant parts of the image elements. For example, a thin colored line in the background could be allowed to obscure part of the shoulders of a person in an image element, but should not cover the person's face. This is particularly important with small detail background objects, which would be obscured by the image elements if they are placed below them in the z-order. Equally we need to prevent them obscuring the important parts of the image elements if they are placed above the image elements in the z-order.
In some embodiments, the background generator 56 uses an “avoidance map” in the creation of the ornamental layers. The avoidance map typically is generated by combining the avoidance map pieces from all the image elements that are selected by the template filler 50. In this process, each avoidance map piece is rotated, scaled, and translated to reflect how its associated image element will be used in the composite. The avoidance map typically is implemented by a grayscale image, the same size as the final composite image, where non-black areas indicate locations which the image elements will occupy in the composite. In some embodiments, the avoidance map consists of a binary black and white image. In other embodiments, the avoidance map additionally includes mid-range grayscale values that signify areas that are occupied by image elements, but which could be partially obscured. In some embodiments, the magnitudes of the grayscale values signify the relative cost of obscuring particular pixels.
The background generator 56 chooses one style from a set of possible styles for the background (
The background generator 56 chooses a dominant color for the background (
The background generator 56 forms a backdrop for the background (
The background generator 56 adds a set of ornamental elements to produce a set of ornamental layers 88 (
The background shapes are selected from a set of available shapes for use with the selected style 82. The number of shapes to be used is different for each background style 82. Each added shape is scaled, rotated and translated to determine the precise appearance in the background. The parameters for scale rotate and translate are determined differently for each style—typically each choice is a weighted random value selected from within some range of possible values defined for the style. Each shape is filled with a colorized texture. The texture is chosen from one of a set of possible textures. The colorization colors are derived from the dominant color. The derived colors may be, for example, lighter or darker shades, harmonizing hues, contrasting hues, etc. Each background shape typically is defined as a separate layer, enabling each shape to potentially be given a cyclic motion definition, as determined by the background style 82. This allows the composite image renderer 14 to cyclically vary parameters of the shape during its appearance on the display 16. Cyclically variable parameters include: position, rotation, intensity, scale. In some embodiments, the background generator 56 renders multiple shapes rendered onto a single image layer or the backdrop layer if there are multiple shapes with a fixed z-order configuration and no independent cyclic motion.
The background detail objects typically include shapes and smaller objects such as lines, curves, text etc. In some implementations, the selected background style 82 defines background details that are to be rendered at a z-order above the image elements. In this case, the background detail objects are positioned so as not to obscure the white areas of the avoidance map. In some implementations, the background style 82 specifies the positioning of background detail objects so as to avoid image elements even when the details are rendered below the image elements in the z-order. In some embodiments, the background generator 56 uses a generate-and-test approach to position the background detail objects based on the avoidance map 92. In this process, the background generator 56 defines a potential detail object by creating the detail object in a temporary image with a known relationship (typically a translation) to the avoidance map 90. The occupied pixels of the detail object are then tested against the avoidance map 90. The proposed shape is rejected if it obscures any white pixel. Depending on the definition of the style and the detail object, the detail object may be accepted if it only obscures grey areas of the avoidance map 90. Like background shapes, each background detail object may be given a cyclic motion, according to the implementation of the selected background style 82. In some cases, it is desirable to prevent multiple detail objects overlapping each other. This can be achieved by rendering the shape of the detail object onto the avoidance map in white. Subsequent use of the avoidance map will then avoid any previously placed background detail objects.
F. Template Filler
The template filler 50 automatically selects image elements that are suitable for a particular template (layout) specification from a pre-loaded collection of image elements. When the template filler 50 is called upon to instantiate a particular template, it searches the database of image elements for a set of image elements that can satisfy the template constraints. In some embodiments, the template filler 50 retains some memory of prior instantiations with a view to controlling the overall flow of template instantiations (e.g., only choose image elements from a particular time or event for a series of templates, or ensure that the image presented are refreshed).
Each template 22 typically specifies parameters that are required of an image element after the image element already has been scaled and translated to fit in the template. For example, the “radius” parameter in the exemplary template specification provided in the attached Appendix indicates the range of face radii that are acceptable for an image element in the parent region. To find an image element for a region that includes a radius constraint, the template filler 50 scales every image element with a suitable number of faces such that all the faces in the image element meet the radius range specification. If the range in the image element is too great, the required scaling may not be possible in which case that image element is not a suitable candidate for the specified region. Once the scaling has taken place, comparisons may be made with the bounding box parameters and the rotation (ipr) parameters. In some embodiments, the image element is not required to fit exactly within the specified boundary of a region; instead, the image element is centered in the region with a width and height that are the maximum allowable for the image element after scaling has taken place.
After the template filler 50 has assigned a set of image elements to the regions of a template, the template is said to be an “instantiated template,” which is a particular arrangement of specific image elements. It is the instantiated template that is scheduled by the scheduler 48 and rendered by the composite image renderer 14 on the display 16. An instantiated template typically is specified by a set of image elements together with their locations (i.e., position of image element centre, rotation angle, and scale factor) in the display area. The locations typically are specified in the coordinate system of the display 16, which typically involves scaling a generic coordinate system to the aspect ratio of the display 16. Although the instantiated template specifies “home” locations for each image element, the actual rendered positions of the image elements varies over time according to the entry and exit behavior ascribed to the layer, and any cyclic motion it is given during the life of the island of stability. As such, it is quite possible for image element layers to overlap for periods of layer motion, though they are generally positioned so as to be all visible.
G. Scheduler and Composite Image Renderer
The scheduler 48 creates a sequence of layers which are rendered by the composite image renderer 14 on the display 16.
As explained above, each image element typically is rendered on its own layer, which has an associated relative z-order location and a motion pattern. Layers have the following characteristics:
In addition to layers for image elements, there are additional layers for background components. The template specification does not directly affect the background layers, except that an avoidance map is generated from the instantiated template. The avoidance map is used during the background creation process, as described above.
In addition to the normal layers used for an island of stability it is sometimes necessary to introduce temporary layers to create some of the desired transitional effects. A temporary layer is basically a clone of an existing layer with its own entry and exit behavior. The temporary layer allows the scheduler to produce the effect of a particular layer that appears to “move up or down through the z-order” this is impossible with the layer mechanism as described. This effect can be achieved by duplicating the layer at the destination z-order position and gradually fading this in, while at the same time fading out the layer at the old z-order position and then deleting the old layer.
The layers which are passed from the scheduler to the renderer are units of fairly complex behavior that include the following components:
Conventional slideshow transitions change from single image to single image using various effects (e.g., fades and wipes), which are independent of the image content. In contrast, a dynamic image collage generation system is capable of using more sophisticated, content-dependent transitions. For example transitions may make use of the meta-data in the media database; for example zooming into a particular face from a photo in the current collage, aligning the eyes to a those of a face from a different photo in the new collage, blending from one face to the other, then moving the new face into position in the new collage. Some embodiments implement transitions that depend on the alignment of the eyes in a face in one of the images used in the old island, with the eyes in a face in one of the images used in the new island. In these embodiments, the scheduler 48 builds the transition from multiple aligned layers.
The composite image renderer 14 and the scheduler 48 interact in a complex manner so as to generate and display a smoothly changing, continuously varied display. The scheduler is responsible for operations including the following:
Some of these operations are computationally expensive.
The composite image renderer 14 generates a sequence of display frames by rendering each of the active layers in their z-order sequence at that frame time. For each frame, each active layer is evaluated based on the current frame time, to generate a set of image rendering parameters for position, scale, rotation, blend, alpha etc. In some implementations, these can be used by graphics hardware to render the image very efficiently. “Evaluation” in this sense means evaluating each of the temporal functions in the layer, modulating the “base” behavior with any “cyclic” variation. The composite image renderer 14 also is responsible for determining how one island will transition to the next. The composite image renderer 14, however, cannot move from one island to the next until the components for the next island have been constructed and loaded into texture memory, which takes an unpredictable amount of time. Thus, in some embodiments, the layers are rendered for a length of time that cannot be determined when they first appear on the display.
Some embodiments overcome the indeterminate scheduling problem by having the scheduler 48 generate a sequence of layers that are processed completely independently by the composite image renderer 14. However, with this approach, it would not be possible to define island N until the data for island N+1 had been loaded, as it is only at that point that we know when island N+1 can start to display, and hence when island N can be stopped. This approach introduces an unpleasant delay in the system, and would require additional memory. In addition, this approach would still lead to the possibility that even though the data for island N+1 was available, the timing for island N+1 could not be established until the data for island N+2 was loaded. This could prevent island N+1 from being started at the time that island N was scheduled to end.
Other embodiments overcome the indeterminate scheduling problem by allowing the end times of layers to be modified while they are actively being rendered. In particular, to allow the layer end time to be initially set as infinite (or a very long time in the future) so that the island N rendering can be started as soon as its data has been loaded. When the scheduler has loaded all the data necessary for island N+1 it can then modify the end times of the layers in island N, create any transition layers necessary to blend between island N and island N+1 and create the layers for island N+1 with infinite end times. The process can then cycle from island to island. The scheduling therefore becomes a process of scheduling transitions from the current island to the next island.
In some embodiments, the process of
In accordance with this modification, some layers are required to be at a known position and display state in order to ensure smooth transitions, given that the current island layers will be at some arbitrary point in their cyclic behavior.
The process of
Embodiments of the dynamic collage generation system 10 may be implemented by one or more discrete modules (or data processing components) that are not limited to any particular hardware, firmware, or software configuration. In the illustrated embodiments, the modules may be implemented in any computing or data processing environment, including in digital electronic circuitry (e.g., an application-specific integrated circuit, such as a digital signal processor (DSP)) or in computer hardware, firmware, device driver, or software. In some embodiments, the functionalities of the modules are combined into a single data processing component. In some embodiments, the respective functionalities of each of one or more of the modules are performed by a respective set of multiple data processing components.
The collage layer generator 12, the composite image renderer 14; and the display 16 may be co-located on a single apparatus or they may be distributed across multiple apparatus; if distributed across multiple apparatus, the collage layer generator 12, the composite image renderer 14, and the display 16 may communicate with each other over local wired or wireless connections, or they may communicate over global network connections (e.g., communications over the internet).
In some implementations, process instructions (e.g., machine-readable code, such as computer software) for implementing the methods that are executed by the embodiments of the dynamic collage generation system 10, as well as the data it generates, are stored in one or more machine-readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile computer-readable memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.
In general, embodiments of the dynamic collage generation system 10 may be implemented in any one of a wide variety of electronic devices, including desktop computers, workstation computers, and server computers.
A user may interact (e.g., enter commands or data) with the computer 120 using one or more input devices 130 (e.g., a keyboard, a computer mouse, a microphone, joystick, and touch pad). In some embodiments, the input devices 130 include a video camera directed at the audience as described in U.S. application Ser. No. ______, filed on even date herewith, and entitled “SYSTEM AND METHOD FOR DYNAMIC GENERATION OF MULTIMEDIA CONTENT” [Attorney Docket No. 200802868-1]. Information may be presented through a graphical user interface (GUI) that is displayed to the user on the display 16 (implemented by, e.g., a display monitor), which is controlled by the composite image renderer 14 (implemented by, e.g., a video graphics card). The computer system 120 also typically includes peripheral output devices, such as speakers and a printer. One or more remote computers may be connected to the computer system 120 through a network interface card (NIC) 136.
As shown in
The embodiments that are described herein are operable to automatically generate a dynamic collage of images (e.g., image elements, such as digital photographs and salient elements extracted therefrom). The collage can be configured to change continuously over time so as to produce a visually appealing ambient experience that is likely to sustain a viewer's interest over long time periods. These embodiments typically have imperfect knowledge of what the user is likely to find interesting. However, by extracting multiple objects of interest from user photos and displaying them together, these embodiments typically are able to provide the user with a rich choice of visual target such that the user is more likely to see at least one thing of interest in the dynamic collage. In some cases, a dynamic collage can leverage the large number of possible variations in the presentation of visual content to reduce the likelihood that the same composition will be presented more than once, thereby ensuring that the presentation of the viewer's image collection always has a refreshing appeal.
Other embodiments are within the scope of the claims.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US2009/032221 | 1/28/2009 | WO | 00 | 7/28/2011 |
