With the recent surge in popularity of digital video, the demand for video compression has increased dramatically. Video compression reduces the number of bits required to store and transmit digital media. Video data contains spatial and temporal redundancy, and these spatial and temporal similarities can be encoded by registering differences within a frame (spatial) and between frames (temporal). The hardware or software that performs compression is called a codec (coder/decoder). The codec is a device or software capable of performing encoding and decoding on a digital signal. As data-intensive digital video applications have become ubiquitous, so has the need for more efficient ways to encode signals. Thus, video compression has now become a central component in storage and communication technology.
Unfortunately, conventional video compression schemes suffer from a number of inefficiencies, which manifest in the form of slow data communication speeds, large storage requirements, and disturbing perceptual effects. These impediments can impose serious problems to a variety of users who need to manipulate video data easily, efficiently, while retaining quality, which is particularly important in light of the innate sensitivity people have to some forms of visual information.
In video compression, a number of critical factors are typically considered including: video quality and the bit rate, the computational complexity of the encoding and decoding algorithms, robustness to data losses and errors, and latency. As an increasing amount of video data surges across the Internet, not just to computers but also televisions, cell phones and other handheld devices, a technology that could significantly relieve congestion or improve quality represents a significant breakthrough.
Systems and methods for processing video are provided to create computational and analytical advantages over existing state-of-the-art methods. A video signal can be processed to create object models from one or more objects represented in the video signal. The object models can be archived. The archived object models can be used as a library of object models for structure, deformation, appearance, and illumination modeling. One or more of the archived object models can be used when processing a compressed video file. The one or more archived object models and a codec can be used to reconstruct the compressed video file. The object models can be used to create an implicit representation of one or more of the objects represented in the video signal.
The object models in the archive can be compared to determine whether there are substantially equivalent object models stored in the archive. The size of the archive can be reduced by eliminating redundant object models that are substantially equivalent to each other. Object models in the archive that are similar can be combined.
A video codec can be used to reconstruct the compressed video file. The object models can be stored separately from the video codec. The object models can be included or bundled with the video codec. A customized codec can be created by grouping several of the object models. The customized codec can be optimized to reconstruct the compressed video file.
The compressed video file can be associated with a group of other compressed video files having similar features. The customized codec can be optimized to reconstruct any of the compressed video files in this group. The group of compressed video files can be determined based on personal information about a user. The personal information about a user can be determined by analyzing uncompressed video files provided by the user. When the uncompressed video files provided by the user are analyzed, reoccurring objects depicted in the uncompressed video files provided by the user can be identified. The reoccurring objects, for example, can be particular human faces or animals identified in the uncompressed video files provided by the user. Customized object models can be created that are trained to reconstruct those reoccurring objects. The customized objects can be used to create a customized codec for reconstructing the compressed video file.
The compressed video file can be sent from one user computer to another. While this compressed video file is being reconstructed, the archived object models can be accessed from a server. The server can be used to maintain and mine the archived object models for a plurality of users. The server can create an object model library. In this way, a video processing service can be provided, where members of the service can store their object models on the server, and access the object models remotely from the server to reconstruct their compressed video files.
The archived object models can be shared among a plurality of user computers in a peer-to-peer network. A request for the compressed video file from one computer in the peer-to-peer network can be received. In response to the request, one of the archived object models can be sent from a different user computer in the peer-to-peer network. Also in response to the request, another one of the archived object models can be sent from yet another computer in the peer-to-peer network. Further in response to the request, another one of the archived object models, or a sub-partitioning of those models can be sent from yet another user computer in the peer-to-peer network. In this way, the archived object models can be maintained and disseminated using a distributed approach.
One or more of object models can be used to control access to the compressed video stream. The object models can be used with a codec to reconstruct the compressed video file. The video file may not be reconstructed or rendered on a user's computer without using one or more of the object models. By controlling access to the object models, access (e.g. playback access) of the compressed video file can be controlled. The object models can be used as a key to access the video data. The playback operation of the coded video data can depend on the object models. This approach makes the compressed video data unreadable without access to the object models. In this way, the object models can be used as a form of encryption and digital rights management. Different quality object models can be used to provide different quality levels of the decompressed video from the same video file. This allows for a differential decoding of a common video file. (e.g. a Standard Definition and High Definition version of the video based on the object model used and a common video file).
One or more of the object models can include advertisements that cause ads to be inserted into the reconstructed video stream upon playback. For example, during reconstruction (e.g. playback) of the encoded video, the models can cause frames that provide advertisement to be generated into the playback video stream.
A software system for processing video can be provided. An encoder can process a video signal to create object models for one or more objects represented in the video signal. An object library can store the object models. A decoder can use a codec and one or more of the archived object models from the object library when reconstructing a coded video file.
The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention.
A description of example embodiments of the invention follows.
Creating Object Models
In video signal data, frames of video are assembled into a sequence of images. The subject of the video is usually a three-dimensional scene projected onto the camera's two-dimensional imaging surface. In the case of synthetically generated video, a “virtual” camera is used for rendering; and in the case of animation, the animator performs the role of managing this camera frame of reference. Each frame, or image, is composed of picture elements (pels) that represent an imaging sensor response to the sampled signal. Often, the sampled signal corresponds to some reflected, refracted, or emitted energy, (e.g. electromagnetic, acoustic, etc.) sampled through the camera's components on a two dimensional sensor array. A successive sequential sampling results in a spatiotemporal data stream with two spatial dimensions per frame and a temporal dimension corresponding to the frame's order in the video sequence. This process is commonly referred to as the “imaging” process.
The invention provides a means by which video signal data can be efficiently processed into one or more beneficial representations. The present invention is efficient at processing many commonly occurring data sets in the video signal. The video signal is analyzed, and one or more concise representations of that data are provided to facilitate its processing and encoding. Each new, more concise data representation allows reduction in computational processing, transmission bandwidth, and storage requirements for many applications, including, but not limited to: encoding, compression, transmission, analysis, storage, and display of the video signal. Noise and other unwanted parts of the signal are identified as lower priority so that further processing can be focused on analyzing and representing the higher priority parts of the video signal. As a result, the video signal can be represented more concisely than was previously possible. And the loss in accuracy is concentrated in the parts of the video signal that are perceptually unimportant.
As described in U.S. application Ser. No. 11/336,366 filed Jan. 20, 2006 and U.S. application Ser. No. 60/881,966, titled “Computer Method and Apparatus for Processing Image Data,” filed Jan. 23, 2007, the entire teachings of which are incorporated by reference, video signal data is analyzed and salient components are identified. The analysis of the spatiotemporal stream reveals salient components that are often specific objects, such as faces. The identification process qualifies the existence and significance of the salient components, and chooses one or more of the most significant of those qualified salient components. This does not limit the identification and processing of other less salient components after or concurrently with the presently described processing. The aforementioned salient components are then further analyzed, identifying the variant and invariant subcomponents. The identification of invariant subcomponents is the process of modeling some aspect of the component, thereby revealing a parameterization of the model that allows the component to be synthesized to a desired level of accuracy.
In one embodiment, the PCA/wavelet encoding techniques are applied to a preprocessed video signal to form a desired compressed video signal. The preprocessing reduces complexity of the video signal in a manner that enables principal component analysis (PCA)/wavelet encoding (compression) to be applied with increased effect. PCA/wavelet encoding is discussed at length in co-pending application, U.S. application Ser. No. 11/336,366 filed Jan. 20, 2006 and U.S. application Ser. No. 60/881,966, titled “Computer Method and Apparatus for Processing Image Data,” filed Jan. 23, 2007.
Segmenter 103 analyzes an image gradient over time and/or space using temporal and/or spatial differences in derivatives of pels. For purposes of coherence monitoring, parts of the video signal that correspond to each other across sequential frames of the video signal are tracked and noted. The finite differences of the derivative fields associated with those coherent signal components are integrated to produce the determined portions of the video signal which use disproportionate bandwidth relative to other portions (i.e., determines the components of interest). In a preferred embodiment, if a spatial discontinuity in one frame is found to correspond to a spatial discontinuity in a succeeding frame, then the abruptness or smoothness of the image gradient is analyzed to yield a unique correspondence (temporal coherency). Further, collections of such correspondences are also employed in the same manner to uniquely attribute temporal coherency of discrete components of the video frames. For an abrupt image gradient, an edge is determined to exist. If two such edge defining spatial discontinuities exist then a corner is defined. These identified spatial discontinuities are combined with the gradient flow, which produces motion vectors between corresponding pels across frames of the video data. When a motion vector is coincident with an identified spatial discontinuity, then the invention segmenter 103 determines that a component of interest (salient object) exists.
Other segmentation techniques are suitable for implementing segmenter 103.
Returning to
The structural object model 107 may be mathematically represented as:
where σ is the salient object (determined component of interest) and SM ( ) is the structural model of that object;
vx,y are the 2D mesh vertices of a piece-wise linear regularized mesh over the object σ registered over time discussed above;
Δt are the changes in the vertices over time t representing scaling (or local deformation), rotation and translation of the object between video frames; and
Z is global motion.
From Equation 1, a global rigid structural model, global motion, pose, and locally derived deformation of the model can be derived. Known techniques for estimating structure from motion are employed and are combined with motion estimation to determine candidate structures for the structural parts (component of interest of the video frame over time). This results in defining the position and orientation of the salient object in space and hence provides a structural model 107 and a motion model 111.
The appearance model 108 then represents characteristics and aspects of the salient object which are not collectively modeled by the structural model 107 and the motion model 111. In one embodiment, the appearance model 108 is a linear decomposition of structural changes over time and is defined by removing global motion and local deformation from the structural model 107. Applicant takes object appearance at each video frame and using the structural model 107 and reprojects to a “normalized pose.” The “normalized pose” will also be referred to as one or more “cardinal” poses. The reprojection represents a normalized version of the object and produces any variation in appearance. As the given object rotates or is spatially translated between video frames, the appearance is positioned in a single cardinal pose (i.e., the average normalized representation). The appearance model 108 also accounts for cardinal deformation of a cardinal pose (e.g., eyes opened/closed, mouth opened/closed, etc.) Thus appearance model 108 AM(σ) is represented by cardinal pose Pc and cardinal deformation Δc in cardinal pose Pc,
The pels in the appearance model 108 are preferably biased based on their distance and angle of incidence to camera projection axis. Biasing determines the relative weight of the contribution of an individual pel to the final formulation of a model. Therefore, perferably, this “sampling bias” can factor into all processing of all models. Tracking of the candidate structure (from the structural model 107) over time can form or enable a prediction of the motion of all pels by implication from a pose, motion, and deformation estimates.
Further, with regard to appearance and illumination modeling, one of the persistent challenges in image processing has been tracking objects under varying lighting conditions. In image processing, contrast normalization is a process that models the changes of pixel intensity values as attributable to changes in lighting/illumination rather than it being attributable to other factors. The preferred embodiment estimates a salient object's arbitrary changes in illumination conditions under which the video was captured (i.e., modeling, illumination incident on the object). This is achieved by combining principles from Lambertian Reflectance Linear Subspace (LRLS) theory with optical flow. According to the LRLS theory, when an object is fixed, preferably, only allowing for illumination changes, the set of the reflectance images can be approximated by a linear combination of the first nine spherical harmonics; thus the image lies close to a 9D linear subspace in an ambient “image” vector space. In addition, the reflectance intensity for an image pixel (x,y) can be approximated as follows.
Using LRLS and optical flow, expectations are computed to determine how lighting interacts with the object. These expectations serve to constrain the possible object motion that can explain changes in the optical flow field. When using LRLS to describe the appearance of the object using illumination modeling, it is still necessary to allow an appearance model to handle any appearance changes that may fall outside of the illumination model's predictions
Other mathematical representations of the appearance model 108 and structural model 107 are suitable as long as the complexity of the components of interest is substantially reduced from the corresponding original video signal but saliency of the components of interest is maintained. Returning to
PCA encoding is applied to the normalized pel data on both sides 232 and 236, which builds the same set of basis vectors on each side 232, 236. In a preferred embodiment, PCA/wavelet is applied on the basis function during image processing to produce the desired compressed video data. Wavelet techniques (DWT) transform the entire image and sub-image and linearly decompose the appearance model 108 and structural model 107 then this decomposed model is truncated gracefully to meet desired threshold goals (ala EZT or SPIHT). This enables scalable video data processing unlike systems/methods of the prior art due to the “normalize” nature of video data.
As shown in
Data Mining Object Models
By archiving these object models (e.g. deformation, structure, motion, illumination, and appearance models), persistent forms of these object models can be determined and reused for processing other video streams. For example, when digital video is imported from a camera, the digital video can be transcoded and the video object archive can be accessed to determine whether any of the object models match. Although this can be done on a frame by frame basis, preferably the portions of the video stream or the entire video stream can be analyzed using batch processing by grouping together similar items. The frames can be analyzed in a non-sequential manner, and a statistical analysis can be performed to determine which object models provide the best fit for coding.
In this way, the present archival system 300 can mine these object models in the object library and analyze object models to identify similar object models. Once the similar object models are identified, the system 300 can capitalize on the redundancy by creating generic object models that can be used over and over again for processing other video. The similarity tends to be based on similar structure, deformation, motion, illumination, and/or appearance.
The object models can be used for subsequent video processing in any number of ways. As discussed in more detail below, the models can be used in a client/server framework, the object models can be bundled into a package with the video codec for use when decoding encoded video file, the models can be used in connection with a personal video service, and the models can be distributed and made available to many users using a distributed system, such as a peer-to-peer network. Also, the processing of the models can occur in a distributed computing network.
Personal Video Processing Service
In the example where the object models are stored on a server, a personal video processing service can be provided.
The system 400 can analyze the object models uploaded from a particular member and determine whether there are redundant object models. If, for example, the member continually transcodes digital video that depicts the same subjects, e.g. the same faces, same pets, etc., it is likely that the same object models will be created over and over again. The system 400 can capitalize on this redundancy by creating a cache of object models that are personal to the user (e.g. a cache of face object models, pet object models, etc.). The system can further capitalize on this redundancy by creating a codec 417 that is customized and personal to that user. The codec 417 can be bundled with the object models 416 that are particular to that user.
By having a substantial amount of members uploaded their models 416 to the server 410, the models can be analyzed to identify common or similar models. The most commonly used or generated models can be tracked. In this way, the system 400 can learn and determine what models 416 are the most likely to be needed, and a codec can be designed to include only the most important object models.
If a user tries to process an encoded video with the codec and the particular model has not been bundled with that codec, the system can access the server 410 to obtain the necessary models from archive 404. The codec may also access the server 410 periodically to update itself with new and updated object models.
As a further embodiment, the encoded videos could be such that the original “conventional” encoding of the video file is accessible on the client node 414. In this case, the advantage of the processing is used for transmitting the video, while more “conventional” compression is used to store the video on the hard disk to facilitate more conventional processing of the video. For instance, if a video editing application wishes to use a different format, then the present inventive method can primarily be utilized during transmission of the video file.
Tuning the Codec
The codec 417 can be tuned to particular types of encoded video data. For example, if the video stream has a reoccurrence of certain objects, a common theme or particular style throughout, than the object models can be reused when reconstructing the entire encoded video file. Similarly, the codec 417 can be optimized to handle these reoccurring objects, such as faces. Likewise, if the video stream is a movie that has certain characteristics, such as a film of a particular genre, such as action film, than it may use similar object models 416 throughout the film. Even where the digital video is a film noir, for example, which is often characteristic of a low-key black-and-white visual style, then particular lighting and illumination object models may be applicable and used when reconstructing the entire encoded version of the movie. As such, there may be common object models (e.g. structure and illumination models) that are applicable to a substantial portion of the encoded movie. These models can be bundled together to create a customized codec.
Sharing Object Models
The object models could also be shared among any number of users. The object models can be stored on a server or in a database so they can be easily accessed when decoding video files. The object models may be accessed from one user computer to another user computer.
In addition to using a client-server framework, object models can be shared and distributed using a peer-to-peer network or other framework. In this way, users can download compressed video files and object models from other users in the peer-to-peer network. For example, if an encoded version of the movie Harry Potter were being downloaded from one system in the peer-to-peer network, to facilitate efficiency the relevant models, or partitions of those models, could be downloaded from other systems in the network.
Digital Rights Management
The process of deploying security schemes to protect access to digital video is long, involved and expensive. Content users want unfettered access to digital content without being required to undergo a burdensome authentication process. One of the most complicated aspects of developing a security model for deploying content is finding a scheme in which the cost benefit analysis accommodates all participants, i.e. the content user, content provider and software developer. At this time, the currently available schemes do not provide a user-friendly, developer-friendly and financially effective solution to restrict access to digital content.
The object models of the present invention can be used as a way to control access to the encoded digital video. For example, without the relevant object models, a user would not be able to playback the video file. The object models can be used as a key to access the video data. The playback operation the coded video data can depend on a piece of auxiliary information, the object models. This approach makes the encoded video data unreadable without access to the object models.
By controlling access to the object models, access to playback of the content can be controlled. This scheme can provide a user-friendly, developer-friendly solution, and efficient solution to restricting access to video content.
Additionally, the object models can progressively unlock the content. With a certain version of the object models, an encoding might only decode to a certain level, then with progressively more complete object models, the whole video would be unlocked. Initial unlocking might enable thumbnails of the video to be unlocked, giving the user the capability of determining if they want the full video. A user that wants a standard definition version would procure the next incremental version of the object models. Further, the user needing high definition or cinema quality would download yet more complete versions of the object model. Both the encoding and the object models are coded in such a way as to facilitate a progressive realization of the video quality commensurate with encoding size and quality, without redundancy.
Processing Environment
In one embodiment, the processor routines 92 and data 94 are a computer program product, including a computer readable medium (e.g., a removable storage medium, such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, hard drives, etc.) that provides at least a portion of the software instructions for the invention system. Computer program product can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection. In other embodiments, the invention programs are a computer program propagated signal product embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network, such as the Internet, or other network(s)). Such carrier medium or signals provide at least a portion of the software instructions for the present invention routines/program 92.
In alternate embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of computer program product is a propagation medium that the computer system may receive and read, such as by receiving the propagation medium and identifying a propagated signal embodied in the propagation medium, as described above for computer program propagated signal product.
Generally speaking, the term “carrier medium” or transient carrier encompasses the foregoing transient signals, propagated signals, propagated medium, storage medium and the like.
While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.
For example, the present invention may be implemented in a variety of computer architectures. The computer network of
The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Some examples of optical disks include compact disk—read only memory (CD-ROM), compact disk—read/write (CD-R/W) and DVD.
A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories, which provide temporary storage of at least some program code in order to reduce the number of times code are retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
Further, in some embodiments, there may be the following advertisement feature.
Embedding Advertisements in the Video Using the Object Models
The object models can be used to cause frames that include advertisements to be inserted into the video stream during playback. In this way, the actual encoded video content would not need to be modified by the advertisements. However, during reconstruction (e.g. playback) of the encoded video, the models can cause frames that provide advertisement to be generated into the playback video stream.
This application is the U.S. National Stage of International Application No. PCT/US2008/000091, filed Jan. 4, 2008, which designates the U.S., published in English, and claims the benefit of U.S. Provisional Application No. 60/881,982 filed Jan. 23, 2007. This application is related to U.S. Provisional Application No. 60/881,966, titled “Computer Method and Apparatus for Processing Image Data,” filed Jan. 23, 2007, U.S. Provisional Application No. 60/811,890, titled “Apparatus And Method For Processing Video Data,” filed Jun. 8, 2006. This application is related to U.S. application Ser. No. 11/396,010 filed Mar. 31, 2006, which is a continuation-in-part of U.S. application Ser. No. 11/336,366 filed Jan. 20, 2006, which is a continuation-in-part of U.S. application Ser. No. 11/280,625 filed Nov. 16, 2005, which is a continuation-in-part of U.S. application Ser. No. 11/230,686, filed Sep. 20, 2005, which is a continuation-in-part of U.S. application Ser. No. 11/191,562, filed Jul. 28, 2005, now U.S. Pat. No. 7,158,680. The entire teachings of the above applications are incorporated herein by reference.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/US2008/000091 | 1/4/2008 | WO | 00 | 7/7/2009 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2008/091484 | 7/31/2008 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
5117287 | Koike et al. | May 1992 | A |
5710590 | Ichige et al. | Jan 1998 | A |
5760846 | Lee | Jun 1998 | A |
5774591 | Black et al. | Jun 1998 | A |
5774595 | Kim | Jun 1998 | A |
5826165 | Echeita et al. | Oct 1998 | A |
5917609 | Breeuwer et al. | Jun 1999 | A |
5933535 | Lee et al. | Aug 1999 | A |
5969755 | Courtney | Oct 1999 | A |
5991447 | Eifrig et al. | Nov 1999 | A |
6044168 | Tuceryan et al. | Mar 2000 | A |
6088484 | Mead | Jul 2000 | A |
6256423 | Krishnamurthy et al. | Jul 2001 | B1 |
6307964 | Lin et al. | Oct 2001 | B1 |
6546117 | Sun et al. | Apr 2003 | B1 |
6574353 | Schoepflin | Jun 2003 | B1 |
6608935 | Nagumo et al. | Aug 2003 | B2 |
6611628 | Sekiguchi et al. | Aug 2003 | B1 |
6625310 | Lipton et al. | Sep 2003 | B2 |
6625316 | Maeda | Sep 2003 | B1 |
6661004 | Aumond et al. | Dec 2003 | B2 |
6711278 | Gu et al. | Mar 2004 | B1 |
6731799 | Sun et al. | May 2004 | B1 |
6731813 | Stewart | May 2004 | B1 |
6738424 | Allmen et al. | May 2004 | B1 |
6751354 | Foote et al. | Jun 2004 | B2 |
6774917 | Foote et al. | Aug 2004 | B1 |
6792154 | Stewart | Sep 2004 | B1 |
6870843 | Stewart | Mar 2005 | B1 |
6912310 | Park et al. | Jun 2005 | B1 |
6925122 | Gorodnichy | Aug 2005 | B2 |
6950123 | Martins | Sep 2005 | B2 |
7003117 | Kacker et al. | Feb 2006 | B2 |
7043058 | Cornog et al. | May 2006 | B2 |
7088845 | Gu et al. | Aug 2006 | B2 |
7158680 | Pace | Jan 2007 | B2 |
7162055 | Gu et al. | Jan 2007 | B2 |
7162081 | Timor et al. | Jan 2007 | B2 |
7164718 | Maziere et al. | Jan 2007 | B2 |
7184073 | Varadarajan et al. | Feb 2007 | B2 |
7352386 | Shum et al. | Apr 2008 | B1 |
7415527 | Varadarajan et al. | Aug 2008 | B2 |
7424157 | Pace | Sep 2008 | B2 |
7424164 | Gondek et al. | Sep 2008 | B2 |
7426285 | Pace | Sep 2008 | B2 |
7436981 | Pace | Oct 2008 | B2 |
7457435 | Pace | Nov 2008 | B2 |
7457472 | Pace et al. | Nov 2008 | B2 |
7508990 | Pace | Mar 2009 | B2 |
7574406 | Varadarajan et al. | Aug 2009 | B2 |
7606305 | Rault | Oct 2009 | B1 |
7630522 | Popp et al. | Dec 2009 | B2 |
7715597 | Costache et al. | May 2010 | B2 |
7788191 | Jebara | Aug 2010 | B2 |
8019170 | Wang | Sep 2011 | B2 |
8036464 | Sridhar et al. | Oct 2011 | B2 |
8065302 | Sridhar et al. | Nov 2011 | B2 |
8068677 | Varadarajan et al. | Nov 2011 | B2 |
8086692 | Sridhar et al. | Dec 2011 | B2 |
8090670 | Sridhar et al. | Jan 2012 | B2 |
8140550 | Varadarajan et al. | Mar 2012 | B2 |
8243118 | Pace | Aug 2012 | B2 |
8259794 | Bronstein et al. | Sep 2012 | B2 |
20010038714 | Masumoto et al. | Nov 2001 | A1 |
20020016873 | Gray et al. | Feb 2002 | A1 |
20020054047 | Toyama et al. | May 2002 | A1 |
20020085633 | Kim et al. | Jul 2002 | A1 |
20020164068 | Yan | Nov 2002 | A1 |
20020196328 | Piotrowski | Dec 2002 | A1 |
20030011589 | Desbrun et al. | Jan 2003 | A1 |
20030058943 | Zakhor et al. | Mar 2003 | A1 |
20030063778 | Rowe et al. | Apr 2003 | A1 |
20030103647 | Rui et al. | Jun 2003 | A1 |
20030122966 | Markman et al. | Jul 2003 | A1 |
20030163690 | Stewart | Aug 2003 | A1 |
20030194134 | Wenzel et al. | Oct 2003 | A1 |
20030206589 | Jeon | Nov 2003 | A1 |
20030231769 | Bolle et al. | Dec 2003 | A1 |
20030235341 | Gokturk et al. | Dec 2003 | A1 |
20040013286 | Viola et al. | Jan 2004 | A1 |
20040022320 | Kawada et al. | Feb 2004 | A1 |
20040107079 | MacAuslan | Jun 2004 | A1 |
20040135788 | Davidson et al. | Jul 2004 | A1 |
20040246336 | Kelly, III et al. | Dec 2004 | A1 |
20040264574 | Lainema | Dec 2004 | A1 |
20050015259 | Thumpudi et al. | Jan 2005 | A1 |
20050185823 | Brown et al. | Aug 2005 | A1 |
20050193311 | Das et al. | Sep 2005 | A1 |
20060029253 | Pace | Feb 2006 | A1 |
20060067585 | Pace | Mar 2006 | A1 |
20060133681 | Pace | Jun 2006 | A1 |
20060177140 | Pace | Aug 2006 | A1 |
20060233448 | Pace et al. | Oct 2006 | A1 |
20070025373 | Stewart | Feb 2007 | A1 |
20070071336 | Pace | Mar 2007 | A1 |
20070153025 | Mitchell et al. | Jul 2007 | A1 |
20070183661 | El-Maleh et al. | Aug 2007 | A1 |
20070185946 | Basri et al. | Aug 2007 | A1 |
20080027917 | Mukherjee et al. | Jan 2008 | A1 |
20080040375 | Vo et al. | Feb 2008 | A1 |
20080101652 | Zhao et al. | May 2008 | A1 |
20080232477 | Wang et al. | Sep 2008 | A1 |
20090040367 | Zakrzewski et al. | Feb 2009 | A1 |
20090067719 | Sridhar et al. | Mar 2009 | A1 |
20090262804 | Pandit et al. | Oct 2009 | A1 |
20090292644 | Varadarajan et al. | Nov 2009 | A1 |
20100008424 | Pace | Jan 2010 | A1 |
20100027861 | Shekhar et al. | Feb 2010 | A1 |
20100049739 | Varadarajan et al. | Feb 2010 | A1 |
20100073458 | Pace | Mar 2010 | A1 |
20100167709 | Varadarajan | Jul 2010 | A1 |
20100278275 | Yang et al. | Nov 2010 | A1 |
20110055266 | Varadarajan et al. | Mar 2011 | A1 |
20110087703 | Varadarajan et al. | Apr 2011 | A1 |
20110182352 | Pace | Jul 2011 | A1 |
20120155536 | Pace | Jun 2012 | A1 |
20120163446 | Pace | Jun 2012 | A1 |
20120281063 | Pace | Nov 2012 | A1 |
20130083854 | Pace | Apr 2013 | A1 |
Number | Date | Country |
---|---|---|
0 614 318 | Sep 1994 | EP |
1 124 379 | Aug 2001 | EP |
1 426 898 | Jun 2004 | EP |
1 779 294 | May 2007 | EP |
2003-253190 | Nov 1991 | JP |
5-244585 | Sep 1993 | JP |
07-038873 | Feb 1995 | JP |
2007-095587 | Apr 1995 | JP |
07-288789 | Oct 1995 | JP |
08-235383 | Sep 1996 | JP |
08-263623 | Oct 1996 | JP |
2001-100731 | Apr 2001 | JP |
2001-103493 | Apr 2001 | JP |
2002-525735 | Aug 2002 | JP |
2004-94917 | Mar 2004 | JP |
2004 356747 | Dec 2004 | JP |
2006-521048 | Sep 2006 | JP |
200521885 | Jul 2005 | TW |
200527327 | Aug 2005 | TW |
WO 9827515 | Jun 1998 | WO |
WO 9859497 | Dec 1998 | WO |
WO 9926415 | May 1999 | WO |
WO 0016563 | Mar 2000 | WO |
WO 0045600 | Aug 2000 | WO |
WO02 102084 | Dec 2002 | WO |
WO 03041396 | May 2003 | WO |
WO 2005055602 | Jun 2005 | WO |
WO 2005107116 | Nov 2005 | WO |
WO 2006015092 | Feb 2006 | WO |
WO 2006034308 | Mar 2006 | WO |
WO 2006055512 | May 2006 | WO |
WO 2006083567 | Aug 2006 | WO |
WO 2006105470 | Oct 2006 | WO |
WO 2007007257 | Jan 2007 | WO |
WO 2007146102 | Dec 2007 | WO |
WO 2008091483 | Jul 2008 | WO |
WO 2008091484 | Jul 2008 | WO |
WO 2008091485 | Jul 2008 | WO |
WO 2010042486 | Apr 2010 | WO |
WO 2010118254 | Oct 2010 | WO |
WO 2011156250 | Dec 2011 | WO |
WO 2012033970 | Mar 2012 | WO |
Entry |
---|
“Bit-Torrent: Introduction”; Jan. 18, 2006 available at http://web.archive.org/web/20060118042451/http://www.bittorrent.com/introduction.html. |
Antoszczyszyn, P.M., et al., “Tracking of the Motion of Important Facial Features in Model-Based Coding,” Signal Processing, 66(2):249-260, (Apr. 30, 1998). |
Fukuhara, T., et al., “3-D Motion Estimation of Human Head for Model-Based Image Coding,” IEEE Proceedings-I, 140(1):26-35, (Feb. 1, 1993). |
International Search Report for International Application No. PCT/US2009/059653, 8 pp., mailed Feb. 2, 2010. |
Written Opinion of the International Searching Authority for International Application No. PCT/US2009/059653, 8 pp., mailed Feb. 2, 2010. |
Extended European Search Report for 06 73 3758.4, dated Mar. 8, 2011 (17 pages). |
Extended European Search Report for 06 74 0318.8, dated May 6, 2011 (14 pages). |
Fischler, M.A., et al., “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Communications of the Association for Computing Machinery, 24(6):381-395 (1981). |
Harris, C., et al., “A Combined Corner and Edge Detector,” Alvey Vision Conference, Proceedings of the Alvey Vision Conference, p. 147 (1988). |
Irani, M., et al., “Detecting and Tracking Multiple Moving Objects Using Temporal Integration,” European Conference on Computer Vision, 282-287 (1992). |
Park, et al., “Qualitative Estimation of Camera Motion Parameters From the Linear Composition of Optical Flow,” Pattern Recognition: The Journal of the Pattern Recognition Society, 37:767-779 (2004). |
Pique, R. et al., “Efficient Face Coding in Video Sequences Combining Adaptive Principal Component Analysis and a Hybrid Codec Approach,” Proceedings of International Conference on Acoustics, Speech and Signal Processing, 3:629-632(2003). |
Schröder, K., et al., “Combined Description of Shape and Motion in an Object Based Coding Scheme Using Curved Triangles,” Proceedings of the International Conference on Image Processing, 2.390-393 (1995). |
Tabatabai, A. J., et al., “Motion Estimation Methods for Video Compression-A Review,” Journal of the Franklin Institute, 335(8): 1411-1441 (1998). |
Wang, Y., “Use of Two-Dimensional Deformable Mesh Structures for Video Coding, Part I—The Synthesis Problem: Mesh-Based Function Approximation and Mapping” IEEE Transactions on Circuits and Systems for Video Technology, 6(6):1051-8215 (1996). |
Wang, Y., “Use of Two-Dimensional Deformable Mesh Structures for Video Coding, Part II—The Analysis Problem and a Region-Based Coder Employing an Active Mesh Representation” IEEE Transactions on Circuits and Systems for Video Technology, 6(6):1051-8215 (1996). |
Shin, J. et al., “Optical flow-based real-time object tracking using non-prior training active feature model,” Academic Press Limited, GB, vol. 11, No. 3, pp. 204-218 (Jun. 1, 2005). |
Cho, J-H., et al., “Object detection using multi-resolution mosaic in image sequences,” Signal Processing. Image Communication, Elsevier Science Publishers, Amsterdam, vol. 20, No. 3, pp. 233-253, (Mar. 1, 2005). |
Gorodinchy, et al., “Seeing faces in video by computers. Editorial for Special Issue on Face Processing in Video Sequences,” Image and Vision Computing, Guilford, GB, vol. 24, No. 6, pp. 551-556 (Jun. 1, 2006). |
Miners, B. W., et al., “Dynamic Facial Expression Recognition Using Fuzzy Hidden Markov Models,” Systems, Man and Cybernetics, 2005 IEEE International Conference on, IEEE, Piscataway, N.J., USA, vol. 2, pp. 1417-1422 (Oct. 10, 2005). |
Dodgson, N. A., “Image resampling,” Technical Report, UCAM-CL-TR-261, ISSN 1476-2986, University of Cambridge, Computer Laboratory, (264 pp.) (Aug. 1992). |
Richardson, I., “Vcodex White Paper: Video Compression Patents,” Vcodex Ltd., pp. 3-6 (2008-2011). |
Jolliffe, I.T., “Principal Component Analysis, Second Edition,” Springer, 518 pp., Apr. 2002. |
Jones, M. and P. Viola, “Fast Multi View Face Detection,” Mitsubishi Electrical Laboratories, Jul. 2003 (10 pp.). |
Viola, P. and M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, vol. 1, pp. 511 518. |
Gunsel, B. et al., “Content based access to video objects: Temporal segmentation, visual summarization, and feature extraction,” Signal Processing, vol. 66, pp. 261 280 (1998). |
Piamsa nga, P. and N. Babaguchi, “Motion estimation and detection of complex object by analyzing resampled movements of parts,” in Proc. ICIP '04, 1 (365 368), Oct. 2004. |
Vidal, R. et al., “Generalized principal component analysis (GPCA)”, in Proc. CVPR '03, 1 (I621-628), Jun. 2003. |
Vidal, R. and R. Hartley, “Motion segmentation with missing data using PowerFactorization and GPCA,” in Proc. CVPR 04, 2 (II-310-316), Jun.-Jul. 2004. |
Huang, R. et al., “Sparse representation of images with hybrid linear models,” in Proc. ICIP '04, 2(1281 1284) Oct. 2004. |
Rong, S. et al., “Efficient spatiotemporal segmentation and video object generation for highway surveillance video,” in Proc. IEEE Int'l, Conf. Communications, Circuits and Systems and West Sino Expositions, 1(580 584), Jun. Jul. 2002. |
Toklu, C. et al., “Simultaneous Alpha Map Generation and 2 D Mesh Tracking for Multimedia Applications,” Proceedings of the International Conference on Image Processing: 1997, (113 116) (Oct. 1997). |
Urban, M., “Harris Interest Operator,” Jan. 28, 2003, http://cmp.felk.cvut.cz/cmp/courses/dzo/resources/lecture—harris—urban.pdf (23 pp.). |
Rehg, J. M. and Witkin, A. P., “Visual Tracking with Deformation Models,” Proc. IEEE Int'l. Conf. on Robotics and Automation, pp. 844-850 (Apr. 1991). |
Kass, Michael, Andrew Witzin, and Demetri Terzopoulos, “Snakes: Active contour Models,” International Journal of Computer Vision (1988). |
Tao, H., et al., “Compression of MPEG-4 Facial Animation Parameters for Transmission of Talking Heads,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, No. 2, pp. 264-276 (Mar. 1999). |
Reinders, M.J.T., et al., “Facial Feature Localization and Adaptation of a Generic Face Model for model-Based Coding,” Signal Processing: Image Communication, No. 7, pp. 57-74 (1995). |
Doenges, P. K., “MPEG-4: Audio/Video and Synthetic Graphics/Audio for Mixed Media,” Signal Processing: Image Communication, No. 9, pp. 433-463 (1997). |
PCT International Search Report, for International Application No. PCT/US2008/000091, dated Sep. 23, 2008, 5 pages. |
PCT International Search Report, for International Application No. PCT/US2008/000092, dated Sep. 23, 2008, 5 pages. |
Notification Concerning Transmittal of the International Preliminary Report on Patentability for PCT/US2009/059653, mailed Apr. 21, 2011 (10 pages). |
Notification Concerning Transmittal of International Preliminary Report on Patentability, in International Application No. PCT/US2008/000091, pp. 9, mailed Aug. 6, 2009. |
Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration, for International Application No. PCT/US2008/000090, 19 pp., mailed Aug. 18, 2010. |
Invitation to Pay Additional Fees and, Where Applicable, Protest Fee, for International Application No. PCT/US2008/000090, mailed Jun. 2, 2010. |
Notification Concerning Transmittal of International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty), for International Application No. PCT/US2008/00090, mailed Sep. 2, 2010. |
Notification and Transmittal of International Search Report and Written Opinion dated Jun. 10, 2013 for PCT/US2013/029297, entitled “Video Compression Repository and Model Reuse”. |
Zhang, et al., “A Novel Video Coding Framework by Perceptual Representation and Macroblock-Based Matching Pursuit Algorithm”, Department of Computer Science and Technology, pp. 322-331 (2007). |
Osama, et al., “Video Compression Using Matching Pursuits”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, No. 1, Feb. 1999. |
Neff, et al., “Matching-Pursuit Based Video Compression”, Department of Electrical Engineering and Computer Science, MPEG Meeting, Mar. 11, 1995. |
Final Office Action from U.S. Appl. No. 13/341,437, dated Jun. 14, 2013. |
Bay, H., et al., “SURF: Speeded Up Robust Features”, ETH Zurich {bay, vangool}@vision.ee.ethz.ch, 1-14 (Date Not Provided), May 13, 2006. |
Lowe, D.G., “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, 1-28 (2004). |
Pati, Y.C., et al., “Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to Wavelet Decomposition”, 27th Annual Asilomar conference on Signals systems and Computers ,1-5 (1993). |
Wiegand, T., et al., “Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560-576 (2003). |
“Series H: Audiovisual and Multimedia Systems: Infrastructure of audiovisual services—Coding of moving video; Advanced video coding for generic audiovisual services”, ITU-T, H.264: 1-657 (2012). |
Garrett-Glaser, J., “Diary of an x264 Developer”, http://x264dev.multimedia.cx/, 1-7 (2008). |
OpenCV Documentation Page, http://docs.opencv.org/ (Retrieved on Dec. 21, 2012). |
Intel Integrated Performance Primitives—Documentation, http://software.intel.com/en-us/articles/intel-integrated-performance-primitives-documentation/ (Retrieved on Dec. 21, 2012). |
Viola, P. and Jones, M.J., “Robust Real-Time Face Detection,” International Journal of Computer Vision, 20(17):138-153 (2004). |
Amit, Yali, 2D Object Detection and Recognition: Models, Algorithms, and Networks, The MIT Press, Cambridge, Massachusetts, pp. 147-149 (Sections 7.3: Detecting Pose and 7.4: Bibliographical Notes and Discussion) (2002). |
Huang, T.S. et al., “Chapter 5: Three-Dimensional Model-Based Image Communication,” Visual Information Representation, Communication, and Image Processing, Editors: Chen, Chang Wen, et al., Marcel Dekker, Inc., New York, New York, pp. 97-117 (1999). |
Notification Concerning Transmittal of International Preliminary Report on Patentability, in International Application No. PCT/US2008/000091, entitled: Object Archival Systems and Methods, pp. 9, mailed Aug. 6, 2009. |
Office Action from U.S. Appl. No. 13/341,437, dated Nov. 20, 2012. |
Office Action from U.S. Appl. No. 12/522,322, dated Dec. 11, 2012. |
Number | Date | Country | |
---|---|---|---|
20100086062 A1 | Apr 2010 | US |
Number | Date | Country | |
---|---|---|---|
60881982 | Jan 2007 | US |