Patent Application
20060139463
The present invention relates to a method of using two or more cameras in an electronic device and to a corresponding electronic device comprising a twin- or multi-camera solution. Particularly it concerns using two or more cameras simultaneously while utilizing a single camera interface and video processing/coding pipeline.
The use of cameras in electronic devices like mobile phones and recently also PDAs has become very common today. At the beginning of the use of this technology in mobile devices such cameras were used for taking still pictures, for example to be used in an MMS. However, with the advance in technology they are more and more also used for capturing video clips.
With some forms of phone housings, particularly flip-type phones, the problem did arise to provide a possibility to take a picture (or video) in the closed position as well as in the open position. Additionally the users wanted to take pictures of themselves while also being enabled to take pictures of the scenery they were looking at, i.e. it should be possible to capture pictures in two directions often being apart from one another by even up to 180°. Also with “rigid” phone housings (“brick type”) the wish did arise to be able to take both pictures of the user of the phone (self-portrait mode) as well as pictures from the user's point of view. To achieve this certain means were used, for example rotatable cameras or camera optics. While this on the one hand seems to be an appropriate solution the amount of mechanically movable parts in a mobile electronic device should on the other hand be kept as low as possible. Such mechanical solutions are not very comfortable to be operated by the user and take up considerable space in the device that is wasted instead of utilizing it for other more useful purposes. Also such constructions with movable cameras are prone to wear when used frequently.
Another solution is therefore to provide mobile phones with two cameras in a single device, for example one pointing in the forward direction to take pictures like with a conventional camera, and another one for taking pictures of the person holding the device. Such twin cameras can be integrated in the housing more conveniently than rotatable cameras or the like, and of course provide for a more robust implementation. However this also presents several disadvantages compared to rotatable or otherwise movable cameras. The costs of an accordingly equipped phone or generally mobile electronic device will apparently be higher as some hardware parts of the cameras have to be provided in duplicate as well. Commonly the manufacturer will simply purchase two independent and substantially complete camera modules to be integrated into the device.
Providing two cameras involves numerous challenges concerning the camera interface, the pre-processing and video coding pipeline, and the physical size, since conventionally the application engine only supports one camera at a time. If two cameras are connected to the system new problems arise in terms of pre-processing and video encoding. Two separate cameras can also be difficult to be housed inside a mobile electronic device due to space restrictions.
The patent application GB 2 394 381 A describes a folding/twisting mobile phone with two housing parts, and comprising two cameras. Only one camera at a time is used, and the switching between the two cameras is performed utilizing a sensor for detecting a positional relation between the two parts of the housing. According to the detected position the operation of the first or second camera is started/stopped.
The patent application GB 2 379 828 A describes a mobile phone with two cameras that is enabled to take photographs from the front and the rear end of the phone, respectively. Here two camera modules are provided back by back, or one camera module and a switchable light deflection device, for enabling the phone to take pictures either from the front or the rear side. The user can switch to either camera view mode to take pictures from the respective side of the phone.
While there are mobile electronic devices known which comprise two cameras they conventionally can not be operated simultaneously to provide two separate video streams, as there is only provided one camera interface and one signal processing path.
Using cameras only for taking (still or single) pictures will assumedly not lead to a big demand of the user to simultaneously take a picture of himself as well as the scene he is looking at, as it will be far more comfortable (concerning focusing, zooming and selecting the wanted part of the picture) to do this successively, i.e. one picture after the other. However with the possibility to record video clips such a demand can arise.
Therefore it is advantageous to provide two cameras in a single mobile electronic device enabling a user to record two video clips or streams simultaneously.
For example the user might wish to record the scene he watches while at the same time recording his reaction to it. Such a feature may be useful for a kind of video-conferencing application, e.g. for simultaneously showing presented sketches or drawings while at the same time displaying the image of the person actually reporting. Being able to simultaneously watch the person speaking can be advantageous not only to for example deaf people, but also improve the understanding of other listeners. Another application in the leisure area may be to record taking a rollercoaster ride and at the same time recording the face of the passenger himself, which will greatly improve the impression that is created compared to showing only what the passenger sees.
The present invention aims at suggesting a method for a more efficient use of two cameras in a single device that can better meet the above discussed demands. A corresponding electronic device is furthermore presented by the invention, which provides two cameras being operable simultaneously, while at the same time keeping the amount of hardware components as low as possible. Both the method and the device of the present invention do not require major changes to the application engine/software platform, camera interface and video processing/coding pipeline of electronic devices to be equipped accordingly.
According to an aspect of the present invention a method for handling video streams of two or more cameras in an electronic device is provided. The electronic device may be a mobile phone, a personal digital assistant (PDA) or a portable or non-portable computer. The method comprises capturing at least first and second video streams by first and second cameras, and integrating said at least first and second video streams into an integrated video stream. This method enables the use of a single camera interface and video processing pipeline for using two or more cameras simultaneously, which enables devices operating correspondingly to be produced with a minimum amount of additional hardware components compared to single-camera implementations. Also this method enables, by minor changes in the application engine or software platform, to handle two or more video streams simultaneously.
In an exemplary embodiment the integrating is performed by generating an integrated video stream, wherein frames of said integrated video stream are alternately provided by frames of said at least first and second video stream, respectively. For example, in the case of two video streams, odd number frames of said integrated video stream may be provided by frames of said first video stream and even number frames may be provided by frames of said second video stream or vice versa. This may of course be extended to n video streams (with n≧2), for example such that every n-th frame of the integrated video stream is provided by a frame of the n-th video stream. Alternately providing frames may also comprise alternately providing sets of frames. For example, first five frames of said integrated video stream may provided by frames of said first video stream and the next five frames may be provided by said second video stream or vice versa. This presents a way of performing the integration in a very straightforward manner. Also the dimensions/aspect ratio of the resulting integrated video stream equal those of the at least two original video streams, which can greatly simplify the further processing. However the sampling rate will in certain embodiments of this method be twice that of the original streams.
In an exemplary embodiment the integrating is performed by generating an integrated video stream, wherein sections of a frame of said integrated video stream are provided by corresponding sections of respective frames of said at least first and second video stream, respectively. In a simple case with two video streams this may for example be performed such that one half of the integrated video stream is provided by frames of the first video stream and the other half by frames of the second video stream. This provides for various suitable implementations, depending on the specific application and the size/aspect ratio of the involved video streams. Even complex integrations may be performed that way, e.g. a kind of picture-in-picture mode.
In an exemplary embodiment the—at least two—sections are arranged adjacent to each other. This includes arranging the frames side by side or even being adjacent in a vertical fashion. Which arrangement is best suited for the specific purpose will mainly depend on the aspect ratios involved. For example two 16:9 video streams will assumedly be integrated to form a 16:18 integrated video stream, rather than 32:9. However this way of performing the integration is easy to implement and still provides a great amount of flexibility to adapt to various applications.
In an exemplary embodiment the integrating is performed by generating an integrated video stream, wherein lines of a frame of said integrated video stream are alternately provided by lines of frames of said at least first and second video stream, respectively. For example, in the case of two video streams, odd lines of each frame of said integrated video stream may be provided by lines of frames of said first video stream and even lines may be provided by lines of frames of said second video stream. Alternately providing lines may also comprise alternately providing sets of lines. Advantages of such integration will be discussed below in this description.
In an exemplary embodiment the method further comprises processing said integrated video stream. As will also be explained later certain processing steps can advantageously be performed on the integrated video stream rather than subsequently to the individual separated output streams.
In an exemplary embodiment the method further comprises separating said integrated video stream into first and second output video streams corresponding to said first and second video streams, respectively, to render said first and second output video streams available for further processing, storing or displaying. If no processing was performed on the integrated video stream the output video streams should be identical to the original video streams. However processing may be performed and the output video streams will still correspond to the original video streams when the separation is performed complementary to the integration step.
According to another aspect of the invention a computer program product is provided, comprising program code means for carrying out the method of the invention, when said program product is run on a computer or network device.
According to another aspect of the invention a computer program product is provided, comprising program code means embodied on a computer readable medium for carrying out the method of the invention, when said program product is run on a computer or network device.
According to another aspect of the invention a computer program product is provided, comprising program code, downloadable from a server for carrying out the method of the invention, when said program product is run on a computer or network device. According to a second aspect of the invention an electronic device is provided, which comprises at least first and second optics, for example lenses or lens systems, at least first and second image sensors, wherein said first image sensor is associated with said first optic and said second image sensor is associated with said second optic. The image sensors are adapted for capturing first and second video streams, respectively. The device further comprises a video integrator unit being adapted for generating an integrated video stream of said at least first and second video streams. Such a device can provide the possibility for a simultaneous use of two or more cameras being constituted by the optics and the image sensors, while keeping the number of hardware components low. As described before such simultaneous use enables various advantageous applications, in the business field as well as for leisure or fun activities. Just one example may be SWIS/video telephony applications.
In an exemplary embodiment the video integrator unit is adapted for generating the integrated video stream such that frames of said integrated video stream are alternately provided by frames or sets of frames of said at least first and second video stream, respectively.
In an exemplary embodiment the video integrator unit is adapted for generating the integrated video stream such that sections of a frame of said integrated video stream are provided by sections or sets of sections of respective frames of said at least first and second video stream, respectively.
In an exemplary embodiment the video integrator unit is adapted for generating the integrated video stream such that lines of a frame of said integrated video stream are alternately provided by lines or sets of lines of frames of said at least first and second video stream, respectively.
In an exemplary embodiment of the invention the device the optics are oriented at 180° to each other. This arrangement of the optics is easy to implement and is of course best suited for the purpose of recording the scenery the user sees and at the same time the user himself. It should though be noted that this is just the most typical solution for the mentioned object; it is within the scope of the invention to use also other orientations like approximately 90°, 45°, 135° or other.
In an exemplary embodiment of the invention the device comprises a video processing unit being adapted for processing said integrated video stream. As will be described later there are certain processing methods that may advantageously be performed even on the still integrated video stream rather than conventionally on the separated output video streams.
In an exemplary embodiment of the invention the device further comprises a camera interface for enabling access to said integrated video stream by an application engine of said mobile electronic device. The invention enables the use of just a single interface rather than requiring complicated and costly solutions with two independent interfaces and a processor handling these interfaces. From the viewpoint of the application engine the integrated video stream may be handled as if it were just a conventional video stream. Everything else may easily be handled in software, such that only slight modifications will be necessary to implement a twin- or multi-camera solution of the present invention.
In an exemplary embodiment of the invention the device further comprises a video separator unit adapted for separating said integrated video stream into first and second output video streams corresponding to said first and second video streams, respectively, to render said first and second output video streams available for further processing, storing or displaying. This provides for various applications or uses of the output video streams. A more detailed explanation will be given in the description of the figures.
In an exemplary embodiment of the invention the device is implemented such that said first and second image sensors are combined into an integral image sensor. For a dual camera solution this is of course a possible manner of implementation. This includes the use of a single image sensor in combination with a suitable optical deflection device (to be explained in more detail below), as well as the use of a “double-sided” sensor, i.e. manufactured of silicon with photo-sensitive areas on both sides.
In an exemplary embodiment of the invention the device further comprises an optical deflection element for directing the views viewed by said first and said second optic to said first and said second image sensor, respectively. In this manner a single image sensor may be utilized, even of a principally conventional type, as will become clear from the drawings.
In an exemplary embodiment of the invention the device further comprises a housing, and said first and second optics are incorporated into a retractable element being settable to at least a first and a second position, wherein said optics are at least partially retracted into said housing in said second position and are operable in said first position. For example the first position may be an “open” position, e.g. wherein the retractable element is folded out, and the second position may be a “closed” position, e.g. wherein the retractable element is folded in. This provides a good mechanical solution for protecting the lenses from dirt and scratching while they are not used. The retractable element may either be folded out manually, or an electric motor may serve for this purpose.
In an exemplary embodiment of the invention the device further comprises a switch being adapted to be actuated when said retractable element is set into said first position and/or when said retractable element is set into said second position. This enables automatic activation or de-activation of camera related software applications. Also the camera hardware may be powered only when needed, to save energy which is usually limited in mobile electronic devices.
Said electronic device may be a mobile electronic device such as a mobile terminal, mobile phone, PDA, etc
The invention will be more fully understood by referring to the appended drawings, which are provided only as illustrative examples and which are not intended to limit the invention to specific embodiments depicted. In the drawings
It is to be noted that the invention can also be applied to more than two video streams and more than two cameras. However, for sake of simplicity the further description will refer to two video streams and cameras, respectively.
Next, in step 210, the two video streams are separated from the integrated video stream again. As they may have been subjected to some processing in optional step 208 this must not necessarily lead to two video streams that are identical to the original two video streams, depending on the processing. However, the separation or extraction is substantially performed in a way that is complementary to the integration process. If, as will be explained later, the integration is performed such that respective frames of video streams are combined into a single frame of the integrated video stream there are also implementations conceivable where the integration/separation processes will not be exactly complementary, depending on the actual application. In steps 212 and 214 the two separated video streams are displayed. It is to be understood that displaying is shown for illustrative purposes only and should not be understood in a limiting fashion. Possible alternatives could for example be storing the two video streams in a memory device, subjecting them to further processing, or even treating the two video streams differently (saving one and displaying the other). The latter may for example be useful for recording the video stream corresponding to the user's point of view onto the memory card of the device, while temporarily buffering the video stream showing the user's face in the RAM memory of the device, to let the user decide if he wants to store it also or discard it. This may also be applied to devices being only able to display one video stream, wherein the other video stream is stored to be displayed later.
Depending on the kind of encoding that is desired the encoding may even, other than depicted here, be performed on the still integrated video stream. This would of course be advantageous as only one encoder would have to be used. However it strongly depends both on the kind of desired encoding method as well as the way the integrated video stream is composed of the original video streams, if encoding of the still integrated video stream is feasible.
Another integration method that is possible within the present invention comprises constituting the integrated video stream by having odd lines being provided by the first video stream and even lines by the second video stream. Although such integration in a kind of “interlaced” fashion may appear more complicated compared with the other methods of the invention there may be special cases wherein the available processing equipment or hardware makes such a method more advantageous though. This particularly concerns possibly required buffering (especially buffer size) and/or the access times of participating memory devices as RAM memories and the like. Here this interlaced way of integrating might help to circumvent certain timing problems that might occur, as the integrating is performed in a line-oriented manner instead of a frame-oriented. Also this might be better suited for cases wherein buffers are too small to buffer whole frames, but are able to buffer frame lines.
It should be noted that, though the above description did focus on the exemplary use case with two cameras and thus two video streams, the inventive concept is of course also applicable to any greater number of cameras and video streams.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/IB04/04265 | Dec 2004 | WO | international |
