This invention relates generally to camera control on a terminal, particularly using gestures received independently of a touch-based interface of the terminal.
It is commonplace for terminals, particularly mobile communications terminals, to comprise one or more cameras.
In the context of this application, a camera is assumed to mean a digital camera capable of generating image data representing a scene received by the camera's sensor. The image data can be used to capture still images using a single frame of image data or to record a succession of frames as video data.
It is known to use video data received by a camera to enable user control of applications running on a terminal. Applications store mappings relating predetermined user gestures detected using the camera to one or more commands associated with the application. For example, a known photo-browsing application uses hand-waving gestures made in front of a terminal's front-facing camera to control how photographs are displayed on the user interface, a right-to-left gesture typically resulting in the application advancing through a sequence of photos.
Some terminals comprise both front- and rear-facing cameras. Prior art applications which run on the terminals enable switching between the cameras by providing a dedicated ‘swap’ icon provided as part of the application's graphical user interface (GUI) which requires the user to touch the button on the GUI.
Disadvantages exist in that developers have to incorporate a dedicated function and icon to effect touch-based control of the camera or cameras via a GUI, e.g. to enable/disable and/or swap between the front and rear cameras. Furthermore, the requirement for users to touch the interface can be problematic in situations where the user cannot hold or touch the terminal, for example when driving or giving a presentation, or where the user is using a rear-facing camera because this camera is on the opposite side to the touch-based interface.
A first aspect of the invention provides apparatus comprising a gesture recognition system configured to detect one or more predetermined user gestures independent of any touch-based interface and to control at least one camera in response to detecting the or each predetermined user gesture.
The apparatus may be configured to disable an enabled camera in response to detecting a predetermined user gesture. The apparatus may be configured to control first and second cameras, wherein the gesture recognition system is further configured to enable a currently-disabled camera in response to detecting the predetermined user gesture.
The gesture recognition system may be configured to receive video data from an enabled camera and to identify from the video data one or more predetermined user gestures. The gesture recognition system may be configured to identify, from the received video data, a gesture represented by a motion vector associated with a foreground object's change of position between subsequent frames of video data, and to compare said motion vector with a set of predetermined reference motion vector to identify a corresponding control command for the at least one camera.
The gesture recognition system may be configured to receive motion signals from a motion sensor and to identify therefrom one or more predetermined user gestures corresponding to said movement. The motion sensor may include at least one of an accelerometer and a gyroscope, the motion signal being generated based on at least one of a change in acceleration and a change in orientation of the apparatus.
The gesture control system may be configured to disable the display of video data from a currently selected camera in response to detection of a predetermined motion gesture and to enable the display of video data from the other, non-selected camera.
A second aspect of the invention provides apparatus, the apparatus having at least one processor and at least one memory having computer-readable code stored thereon which when executed controls the at least one processor:
A third aspect of the invention provides a method comprising:
The outputted command may be configured to disable a currently-enabled camera. The outputted command may be configured to enable a currently-disabled camera.
Receiving gestural data may comprise receiving video data from the at least one camera and identifying from the video data one or more predetermined user gestures. Receiving gestural data may further comprise identifying a motion vector associated with a foreground object's change of position between subsequent frames of video data, and comparing said motion vector with a set of predetermined reference motion vectors to identify a corresponding control command for the or each camera. Receiving gestural data may comprises receiving a signal from a motion sensor provided on the device, the signal being representative of movement of the device, and identifying therefrom one or more predetermined user gestures corresponding to the sensed movement. The signal may be received from at least one of an accelerometer and gyroscope, the signal being generated based on at least one of a change in acceleration and a change in orientation of the device.
The method may comprise, in response to detection of a predetermined motion gesture, disabling display of video data from a currently selected camera and enabling the display of video data from a non-selected camera.
Another aspect provides a computer program comprising instructions that when executed by a computer apparatus control it to perform any method above.
Another aspect provides a portable device comprising any of the apparatus above.
A further aspect of the invention provides a non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method comprising:
Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
a and 8b are perspective views of the mobile terminal shown in
Referring firstly to
The front camera 105a is provided on a first side of the terminal 100, that is the same side as the touch sensitive display 102. The rear camera 105b is provided on the opposite side of the terminal.
The memory 112 may be a non-volatile memory such as read only memory (ROM) a hard disk drive (HDD) or a solid state drive (SSD). The memory 112 stores, amongst other things, an operating system 126 and may store software applications 128. The RAM 114 is used by the controller 106 for the temporary storage of data. The operating system 126 may contain code which, when executed by the controller 106 in conjunction with RAM 114, controls operation of each of the hardware components of the terminal.
The controller 106 may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors.
The terminal 100 may be a mobile telephone or a smartphone, a personal digital assistant (PDA), a portable media player (PMP), a portable computer or any other device capable of running software applications and providing audio outputs. In some embodiments, the terminal 100 may engage in cellular communications using the wireless communications module 122 and the antenna 124. The wireless communications module 122 may be configured to communicate via several protocols such as GSM (Global System for Mobiles), CDMA (code division multiple access), UMTS (universal mobile telephone system), Bluetooth and IEEE 802.11 (Wi-Fi).
The display part 108 of the touch sensitive display 102 is for displaying images and text to users of the terminal and the tactile interface part 110 is for receiving touch inputs from users.
As well as storing the operating system 126 and software applications 128, the memory 112 may also store multimedia files such as music and video files. A wide variety of software applications 128 may be installed on the terminal including web browsers, radio and music players, games and utility applications. Some or all of the software applications stored on the terminal may provide audio outputs. The audio provided by the applications may be converted into sound by the speaker(s) 118 of the terminal or, if headphones or speakers have been connected to the headphone port 120, by the headphones or speakers connected to the headphone port 120.
In some embodiments the terminal 100 may also be associated with external software application not stored on the terminal. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications can be termed cloud-hosted applications. The terminal 100 may be in communication with the remote server device in order to utilise the software application stored there. This may include receiving audio outputs provided by the external software application.
In some embodiments, the hardware keys 104 are dedicated volume control keys or switches. The hardware keys may for example comprise two adjacent keys, a single rocker switch or a rotary dial. In some embodiments, the hardware keys 104 are located on the side of the terminal 100.
Specifically, the command map 142 stores one or more commands, which, when executed by the controller, causes switching of one or both cameras 105a, 105b between enabled and disabled modes, as well as swapping control between the cameras so that when one camera is enabled, the other is disabled. In this sense, enabling a particular one of the cameras 105a, 105b means making the controller 106 configured to receive image or video data from the enabled camera for output to the display 108 and also to enable capture of the transferred image or video data using a camera application (not shown) handling aspects such as zoom, capture and storage on the memory 112.
As will be described in greater detail below, the gesture detection application 140 identifies gestures from, in a first embodiment, either of the front and rear cameras 105a, 105b and, in a second embodiment, the motion sensing accelerometers and/or gyroscopes 130. It will therefore be appreciated that camera control can be achieved independently of the touch sensitive display 120 and indeed of other hard keys provided on the terminal 100.
Referring to
In a subsequent step 4.4, if a received gesture is matched with a reference gesture stored in the command map 142, it is mapped to its associated command in step 4.5 which is then executed in step 4.6 by the controller 4.6 to perform a predetermined camera function.
A first embodiment will now be described in greater detail with reference to
Referring to
Referring to
Referring to
a and 8b show an example of how the gesture detection application 140 can be advantageously employed.
Referring to
When the user is making a presentation, they may initially enable the front camera 105a which is either the default camera or, if not, by waving their hand from right-to-left to cause the gesture detection application 140 to switch camera control from the rear camera 105b to the front camera. With the front camera 105a enabled, the user can operate the presentation application using the appropriate hand gestures to scroll through the slides. If at any time the user wishes to move in front of the terminal 100 to highlight something on the projector screen 174 by way of hand gestures, they will need to enable the rear camera 105b. Again, they may switch camera control using a right-to-left swipe gesture before the front camera 105a.
Referring to
This usage example demonstrates a further advantage in being able to control one or both cameras 105a, 105b remotely of the terminal 100 in that the user avoids disturbing the position of the terminal which should remain stationary in use; otherwise the terminal will need to be re-aligned with the projector screen 174.
A further point to make is that, when one of the cameras 105a, 105b is enabled, there is a relatively large power consumption. In a typical device, an enabled front camera 105a may typically run down a fully charged 1000 mAh battery in about an hour. So, the ability to switch the cameras 105a, 105b off when they are not needed is advantageous to save power and can be easily effected in the present embodiment using the relevant hand waving gesture. Consider the situation where the terminal 100 is connected to a holder on a car dashboard and the driver is using the front camera 105a to hold a hands-free conference call. If battery power is running low, the driver may wish to switch off the camera 105a and use voice-only communications. The driver avoids the need to locate and physically touch the relevant ‘off’ button terminal 100 by simply making the appropriate gesture in the camera's field-of-view. Switching the front camera 105a back on may employ detection of a different gesture, perhaps based on motion, as will be introduced in the second embodiment described below.
A second embodiment will now be described with reference to
Referring to
In the present use example, movement corresponding to a wrist turnover action, indicated in
Referring to
In general, the second embodiment avoids conflict problems that may arise in the first embodiment where both the gesture detection application 140 and a proprietary application use gestural information detected from one or more of the cameras 105a, 105b. Here, camera control is effected using a different set of movement sensors.
A further practical use of the second embodiment will now be described. It will be appreciated that, in general, the rear camera 105b of a communications terminal will have a greater resolution and frame rate than that of the front camera 105a which is on the same side as the touch sensitive display 102. Therefore, use of the rear camera 105b may be preferred over the front camera 105a for certain tasks involving hand-movement detection, e.g. to control a proprietary application. Also, hybrid use of both front and rear cameras 105a, 105b may be preferred to differentiate between similar gestures or between basic and advanced gestures. Therefore, using a wrist turning action, as indicated in
Using both cameras 105a, 105b in this way will achieve greater recognition accuracy than just one of the cameras, particularly for handwaving or ‘hovering’ recognition applications.
It will be seen that the devices described above provide for user control of one or more cameras through gestures independent of any touch-based interface, that is without the use of keys or a touch-screen. This means that application developers do not have to incorporate dedicated command buttons or icons into their GUI code to cater for touch-based camera control. Further, the or each camera can be controlled remotely from the terminal in certain situations.
It will be appreciated that the above described embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present application.
Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.