Embodiments of the invention will now be described, by way of example only, with reference to the drawings, of which:
There will now be described by way of example the best mode contemplated by the inventors for carrying out the invention. In the following description numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent however, to one skilled in the art, that the present invention may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the present invention.
In overview a composite still image such as a digital photograph is composed from multiple time varying still image instances, for example a sequence of individual still images captured sequentially in time. The multiple still images are encoded using an encoding scheme such as mpeg. As discussed in more detail below, a compression step in mpeg encoding comprises encoding one or more still image instances by identifying and storing only the differences between those instances and a reference image in the form of difference information associated with a delta or motion vector. In addition mpeg encoding in some instances allows identification of individual aspects of an image as objects. The composite still image can then be composed from one of the still image instances for example the reference image instance or a difference image instance together with the difference information. For example if it is identified that an aspect of the scene changed between instances undesirably then that change (together with all other changes) will be represented as difference information such that the difference information can simply be undone. As a result the composite image will correspond to say, the difference image in all aspects except for the non-desired change which can simply be undone.
As a result interval photography, where a sequence of instances are captured over a time interval, with the particular instant determined by various means such as regular or periodic capture as in a movie or triggered captures as used in stop-motion or time lapse photography can be used to produce one or more composite still images that is a synthesis of the interval. In particular areas that it is desired to change can be simply identified and dealt with simply by undoing the difference information or, indeed, applying difference information as appropriate. Furthermore these differences can be identified simply by identifying all difference information corresponding to a selected area or mpeg object where mpeg supports such objects.
The mpeg standards are well known to the skilled reader and described at www.chiariglione.org/mpeg such that detailed description is not required. However for the purposes of clarity certain basic principles will now be set out.
According to the mpeg standard mpeg 2 a sequence of images or frames are encoded for example of a time-varying scene. The sequence of images is encoded into a sequence of encoded frames. The first frame the sequence will typically be an intra-frame or I-Frame which contains all of the information necessary to reconstruct a complete image, acting as a reference image instance. Subsequent frames in the sequence will typically be predictive (P) or bi-directional (B) frames which contain only difference information between one or more other frames.
Much of the compression available from the mpeg standard arises because aspects of a scene do not necessarily change significantly or at all between multiple image instances captured at intervals during a time-varying scene. As a result subsequent (or indeed preceding) frames or image instances termed here difference image instances to the reference frame can simply replicate those aspects of the image common with the reference frame such that only differences between the frames need be recorded. In particular according to the mpeg standard the reference frame is segmented into blocks of pixels and each block is encoded and compressed in an appropriate manner such as Discrete Co-sign Transform (DCT). Difference image instances such as P- or B-frames are similarly encoded using DCT into blocks and the blocks are compared with a reference image instance or another difference image instance. If the blocks are identical then the compared frame does not need to record the detailed pixel information but can simply replicate the corresponding block from the reference frame. If the blocks differ slightly then only the differences need be recorded as difference information in the difference image instance for example as a “delta”. If an object has moved between image instances then movement of the corresponding blocks from the reference frame can be encoded as a “motion vector” for example in terms of the instance and direction moved by the block. Of course if a block in a P-frame or B-frame cannot be identified, say, in the I-frame then all of the pixel information must be encoded in the B- or P-frame. As a result the mpeg scheme comprises an encoding scheme which not only performs the initial DCT encoding but also encodes differences between a difference image instance and a reference frame or image instance which may itself be a difference image instance as difference information associated with a motion vector or delta.
According to versions of mpeg such as mpeg 4, in addition specific elements at a scene can be identified and encoded as mpeg objects or “video objects” such that video scenes can be composed of multiple, independent varying objects.
The method described herein, according to one embodiment, can be further understood with reference to
At step 600 in
At step 602, the differences between the images are obtained as part of standard mpeg encoding. The images can be immediately or subsequently stored in the mpeg form. Where, for example, image instance 400 is the reference image instance or I-frame and image instance 402 is the difference image instance or B-frame or P-frame the differences are encoded as deltas or motion vectors as shown at 404. In particular it can be seen that difference image instance 402 is divided into blocks (only two of which, each of which may represent multiple blocks in practice, are shown fully for ease of understanding) 404, 406. It will be seen that at 402 the difference information is recorded as Δ1, and Δ2. In addition, objects (for example each eye) may be identified and encoded.
Reverting to
It will further be noted that identification by the selector or the desired change is achieved very simply. For example where the difference image instance is represented on a GUI such as a camera or computer screen, the selector can identify the area where it is desired to undo a change in any appropriate manner for example by defining the relevant area with a mouse or on a touch screen. If the selector then also identifies the reference image containing the desired replacement area then all difference information in the selected area on the difference image instance can be automatically identified for example by virtue of the coordinates of the corresponding encoded block or blocks, and the difference information simply undone.
Alternatively or in addition, the user may identify the, where supported by the mpeg version, video object requiring changes the system then undoing/applying the corresponding difference information.
Indeed the difference information can be used to identify to the selector or editor what has changed and the editor can then select the version of the object that is desired, where the undesired difference is not apparent. The composite image or selected version can then be constructed by using or omitting deltas from the sequence to construct the final still image which hence does not represent a specific point in time but a synthesis of an interval of time. Individual aspects may not have time coherence as events that occurred out of order can be represented on a single composite image. It will further be recognised that multiple changes can be selected between multiple images and the deltas applied such that a final image may be a composite of multiple individual image instances.
According to another embodiment of the method, motion or other effects can be depicted by applying repeated deltas, as can be seen from
It will be appreciated that the method described herein can be implemented on any appropriate apparatus for example a digital camera or computer apparatus configured to encode multiple image instances according to the mpeg or similar standard. Such an apparatus is shown schematically as 900 in
Accordingly when multiple image instances are captured they can be processed according to the mpeg standard and the difference information encoded. The user can then, for example, view the desired images on a GUI, select the desired images and changes and store, display or print the images as appropriate.
It will be appreciated that the method steps can be implemented in any appropriate manner for example in software or hardware and that appropriate code for manipulation of the images and composition of a composite still image in the manner described above can be programmed in any suitable manner as will be apparent to the skilled reader without the requirement for detailed discussion here.
As a result of the arrangement described herein a simple image composition system is provided in which the editor is not required to identify objects which have boundaries, for example but can simply rely on identification of changes and in which the editor is not limited to selection from a set of instance; fictional instance can be constructed as desired.
It will be appreciated that the approaches described herein can be implemented using any version of the mpeg standard (mpeg-1, mpeg-2, mpeg-4, mpeg-7 or mpeg-21) or indeed any other encoding scheme in which differences between image instances are encoded as difference information for example the H.26* standard promulgated by the International Telecommunication Union (ITU) and described at www.itu.int. Any appropriate criteria for the creation of the synthesis can be adopted for example desired/selected objects, average intensities, depiction of motion such that video recording technology e.g. mpeg can be used advantageously to identify and select objects, intensities, motion and so forth to compose the still images.
Number | Date | Country | Kind |
---|---|---|---|
06300499.8 | May 2006 | EP | regional |