This patent application relates to devices and methods for interfacing with a user.
In mobile devices such as a smart phone, a camera phone or a tablet computer it is known to display a live video of an object 110 (such as a business card) in the real world on a screen 101 of a mobile device 100 (see
It is further known to use a technology commonly known as augmented reality, to overlay content (most often 3D content) on a video being displayed by such a mobile device. The content can be displayed stationary relative to a portion of an image on the screen indicative of an object in the real world. For example, if the object in the real world is a saucer, a virtual object in the form of a cup can be overlaid on the saucer (“target”) in the image on the screen. Movement of the real-world saucer relative to the camera can result in movement on the screen of both the cup and the saucer together (kept stationary relative to one another).
In mobile devices of the type described above, when a user is interacting with augmented reality by reaching into a real world scene to move a virtual object displayed on the screen, if a user wants to issue a command to the mobile device, the user needs to use a normal interface (e.g. touch screen, joystick or microphone) of the mobile device. The inventor of the current patent application believes that use of the normal interface of a mobile device takes time and adds additional mental load on the user, which is a drawback of certain prior art.
An article entitled “Visual Code Widgets for Marker-Based Interaction” by Michael Rohs describes visual codes (two dimensional barcodes) that can be recognized by camera-equipped mobile devices, in real time in a live camera image. Visual code equipped widgets make it possible to design graphical user interfaces that can literally be printed on paper or shown on large-scale displays. Interaction typically takes place as follows: the user finds a visual code widget, for example in a magazine. She starts a recognizer application on her phone or PDA and aims at the widget. The widget appears on the device screen in view finder mode and is updated in real time as the user moves the device relative to the widget. The state of the widget is superimposed over the camera image. Menus are widgets that trigger a function upon selection of a menu item. Pen-based input can be used for selection of the menu item. For devices without pen-based input, pressing the joystick button can take a picture so the camera image freezes, and the user has the opportunity to cycle through the menu selection using the joystick. One more click submits the selected menu item. Accordingly, it appears that menus of the type described by Michael Rohs are useful for interfacing a user with objects that are either static in the real world or too heavy for the user to move in the real world.
An article entitled “Mixed Interaction Spaces—a new interaction technique for mobile devices” by Hansen et al. describes Mixed Interaction Space (MIXIS) which uses the space surrounding a mobile device for its input. The location of a mobile device is tracked by using its built-in camera to detect a fixed-point in its surroundings. This fixed-point is then used to determine the position and rotation of the device in the 3D space. The position of the mobile phone in the space is thereby transformed into a 4 dimensional input vector. In one example, movement of the mobile device with the user's head as the fixed-point is mapped to actions in a graphical user interface on the device. MIXIS eliminates the need to use two dimensional barcodes or visual codes as described above. However, moving a relative heavy device like a tablet, to generate input vectors, can lead to fatigue.
In several aspects of various embodiments, an electronic device (such as a mobile device) displays on a screen of the device, a live video captured by a camera in the device. While the live video is being displayed, the device checks if a first predetermined condition is satisfied. When the first predetermined condition is satisfied the device displays a menu on the screen. The menu includes multiple menu areas, one of which is to be selected.
In certain embodiments, while the menu is being displayed on the screen, the device checks if a second predetermined condition is satisfied, e.g. by a movement of a predetermined object in real world outside the device. When the second predetermined condition is satisfied, the device displays on the screen at least an indication of a menu area as being selected from among multiple menu areas in the displayed menu.
Therefore, a user of the device can easily select a menu area in a menu, by simply moving a predetermined object in the real world. Accordingly, in some embodiments, the user does not need to touch the screen to make a selection. Instead, in several such embodiments, the user holds a mobile device in one hand and moves the predetermined object in the other hand, to make a selection of a menu area in a menu displayed by the mobile device.
Various embodiments are implemented as a system including a camera and a screen operatively connected to one another. The system includes means for checking if a first predetermined condition is satisfied, while a live video captured by the camera is being displayed on the screen, means for displaying on the screen at least a menu including multiple menu areas when at least the first predetermined condition is satisfied, means for checking if a second predetermined condition is satisfied by a movement of a predetermined object in real world, while the menu is being displayed on the screen and means for displaying on the screen at least an indication of a menu area among the menu areas as being selected, when at least the second predetermined condition is satisfied.
Several embodiments are implemented as a mobile device that includes a camera, a memory operatively connected to the camera, a screen operatively connected to the memory to display a live video captured by the camera, and one or more processors operatively connected to the memory. The memory includes instructions to the one or more processors, including instructions to check whether a first predetermined condition is satisfied while the live video is being displayed on the screen, instructions to display on the screen at least a menu including multiple menu areas when at least the first predetermined condition is found to be satisfied by execution of the instructions to check, instructions to check whether a second predetermined condition is satisfied by a movement of a predetermined object outside the mobile device, while the menu is being displayed on the screen and instructions to display on the screen at least an indication of a menu area as being selected when at least the second predetermined condition is satisfied. Certain embodiments are implemented as a non-transitory computer readable storage medium that includes the just-described instructions (i.e. instructions described in the current paragraph) for execution by one or more processors of a mobile device or other such electronic device.
It is to be understood that several other aspects of the embodiments will become readily apparent to those skilled in the art from the description herein, wherein it is shown and described various aspects by way of illustration. The drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
In several aspects of various embodiments, an electronic device and method use a camera on a rear side of the electronic device (an example of which is mobile device 300 in
During the display of live video of the real world environment, handheld electronic device 200 checks if a first predetermined condition is satisfied (see act 203 in
A predetermined object, whose proximity is being detected in act 203 may or may not contain electronics, depending on the embodiment. Illustrative examples of a real world object that is sufficiently small and light to be held in a human hand and which can be used in many embodiments as a predetermined object to satisfy a predetermined condition of the type illustrated in act 203 include: (1) business card, (2) credit card, (3) pencil, (4) paper clip, (5) soda can, (6) spoon, (7) key, (8) mouse, (9) cell phone, (10) remote control, or (11) toy. Therefore, any such predetermined object, whose proximity is detected in act 203 is not necessarily a traditional input device, such as a wireless mouse, although a wireless mouse can be used as the predetermined object in some embodiments of the type described herein.
Other embodiments of act 203 may perform other tests to additionally or alternatively check whether a first predetermined condition is satisfied, e.g. 1) whether a voice command is received or 2) whether a test is satisfied for proximity of one predetermined object to another predetermined object. For example, a distance in an image in the live video between a credit card and a business card, of less than 1 cm satisfies the first predetermined condition of act 203 of some embodiments. Depending on the embodiment, handheld electronic device 200 may check either a single condition or multiple conditions in act 203, such as (a) presence of a predetermined object in an image of live video and (b) presence of a specific pattern on the predetermined object that was found to be present as per (a). Therefore, in one example of such embodiments, a first predetermined condition is satisfied only when a credit card is detected in live video that is displayed by electronic device 200 and furthermore when the credit card carries a specific two-dimensional bar code (e.g. the credit card 's 2D bar code may uniquely identify, for example, a specific financial institution that issued the card).
When the first predetermined condition is satisfied in act 203, handheld electronic device 200 displays a menu on its screen (see act 204 in
In certain embodiments, while the menu is being displayed on the screen, handheld electronic device 200 checks if a second predetermined condition is satisfied during such display (see act 205 in
When the second predetermined condition is found to be satisfied in act 205, the handheld electronic device 200 displays on its screen at least an indication of a menu area as being selected, from among multiple menu areas in the displayed menu (see act 206). Thereafter, in act 207, handheld electronic device 200 performs an action that is associated with menu area that was selected and optionally erases the displayed menu (see act 203D). In some embodiments, when the second predetermined condition is not satisfied in act 203, handheld electronic device 200 returns to performing act 201 (described above).
As noted above, an object whose proximity is detected in act 203 is predetermined, e.g. the object is identified to handheld electronic device 200 by a user ahead of time, prior to acts 201 and 202. This predetermined object is detected within the live video being captured as per act 203, in some embodiments by a method illustrated in
Thereafter, in act 203C, handheld electronic device 200 checks if the distance is within a predetermined threshold (e.g. Zthresh illustrated in
Handheld electronic device 200 described above in reference to
Mobile device 300 is configured to display on screen 301, a predetermined menu 304 formed by four drop shaped areas (such as areas 304I and 304J of screen 301 in
In several embodiments, a threshold distance Zthresh (see
Typically, in several embodiments, threshold distance Zthresh is predetermined to be a number that is of the same order of magnitude as a dimension (e.g. width W) of mobile device 300, which is a hand-held device in such embodiments. In situations wherein Z first <Zthresh, object 302 is within the vicinity of mobile device 300, such that a combination of camera 310 and screen 301 in device 300 operate together as a magnifying lens. Configuring device 300 to operate as a magnifying lens while displaying menu 304, by selection of an appropriate value of threshold distance Zthresh, enables a user of device 300 to perform movements on object 302 in the real world that are small relative to corresponding movements of image 309 (also called “target”) on screen 301. Therefore, a user can make a small movement of object 302 by moving the user's right hand 303R in the real world in order to make a corresponding movement of icon 308 sufficiently large to cause a menu area on screen 301 to be selected.
For example, as described below, a movement dX along the negative X direction in
Depending on the embodiment, movement dS is predetermined to be smaller than an X or Y dimension of screen 301, e.g. dS<W/3 wherein W is the width of device 300. Moreover, dS is predetermined to be large enough to enable the user to make a selection of a menu area 304J from among multiple menu areas of menu 304 displayed on screen 301, e.g. dS>B/2 wherein B is the distance between two menu areas 343 and 344 (see
The specific value of Zthresh that is predetermined in various embodiments depends on multiple factors, such as an angle (also called “opening angle”) (e.g. 60° degrees in
In some aspects of the described embodiments, software 320 (also called “app”) of mobile device 300 displays menu 304 stationary relative to screen 301, and icon 308 is displayed stationary relative to image 309 (or a portion thereof) captured from predetermined object 302. In some embodiments of software 320 (also called “application software” or simply “app”), menu 304 is rendered on screen 301 by invoking augmented reality (AR) functionality of mobile device 300 using menu data 330 (
Referring to
In several aspects of the described embodiments, selection interface 326 uses augmented reality (AR) functionality to move icon 308 automatically in response to movement of image 309. In several such embodiments, when movement of predetermined object 302 results in a position of the icon 308 touching or overlapping an area of menu 304, the second predetermined condition is satisfied. When the second predetermined condition is satisfied, selection interface 326 displays an indication on the screen 301 that the menu area is selected (e.g. by highlighting the menu area).
For example, menu area 304J of menu 304 is highlighted (as shown by cross-hatch shading in
Referring back to
Some embodiments of the type described above are implemented as illustrated by an example in
Pixel values for menu 340 and icon 348 are generated by software instructions of a rendering module 351 that are stored in memory 319 and executed by one or more processors 306 in the normal manner. When executing the instructions of rendering module 351, processor 306 receives input data from menu interface 325, which in turn uses menu data 331-334 to identify the shapes and positions of corresponding menu areas 341-344. Menu interface 325 of several embodiments typically includes a checking module 325C to perform act 203 as described above in reference to acts 203A-203D shown in
In addition to rendering module 351, memory 319 of several embodiments of the type illustrated in
In addition to modules 351 and 352 described in the preceding two paragraphs, memory 319 of several embodiments of the type illustrated in
Accordingly, in some embodiments, checking module 325C constitutes the means for checking if a predetermined condition is satisfied as per act 203, while a live video captured by the camera 310 is being displayed on the screen 301. In such embodiments, checking module 326C constitutes means for checking if another predetermined condition is satisfied e.g. as per act 205 by movement of the predetermined object 302 in real world, while the menu is being displayed on the screen. Depending on the embodiment, checking module 326C may check on movement of predetermined object 302 in the X-Y plane to trigger selection of a menu area within a displayed menu, or checking module 326C may check on movement of predetermined object 302 in the Z direction to trigger display of another menu.
Moreover, in such embodiments, rendering module 351 renders on screen 301 as per act 204, a first display of a menu comprising a plurality of menu areas when at least the first predetermined condition is satisfied. In some embodiments, rendering module 351 includes in the first display a predetermined icon 348 overlaid on a portion of image 309 of predetermined object 302 in the first display. In such embodiments, rendering module 351 moves the predetermined icon on the screen 301 in response to a signal indicative of movement of predetermined object 302 in X-Y plane in the environment in real world. Subsequently, rendering module 351 may render on screen 301 as per act 206, a second display of an indication of a menu area (in the plurality of menu areas) as being selected, when another predetermined condition is satisfied. Between the first and second displays, rendering module 351 may render several intermediate displays showing movement of an icon between menu areas. Alternatively or additionally, rendering module 351 may render on screen 301 a second menu comprising a second set of menu areas, to replace a first menu previously included in the first display, e.g. in response to another signal indicative of movement of predetermined object 302 in the Z direction.
The above described modules 351-353 are together included, in some embodiments, in software instructions 350 stored in memory 319 that when executed by processor(s) 306 implement augmented reality (AR) functionality. Note, however, that in alternative embodiments such augmented reality (AR) functionality is implemented by specialized circuitry in hardware of mobile device 300. In still other embodiments, such augmented reality (AR) functionality may be implemented external to mobile device 300, e.g. in an external computer (not shown) accessible wirelessly by mobile device 300 (e.g. via a cell phone network). Therefore, a specific manner in which modules 351-353 are implemented is not a critical aspect of several embodiments.
Accordingly, as shown in
As noted above, in several embodiments of the type described herein, the user's left hand 303L is used to hold mobile device 300 steadily while performing movements on object 302. Even as object 302 is being moved by right hand 303R, the user's left hand 303L steadily holds device 300 which enables the user 303 to focus their eyes on screen 301 more easily than its opposite interaction. In the just-described opposite interaction which is implemented in some alternative embodiments, the user 303 keeps object 302 steady in their right hand 303R while moving device 300 using their left hand 303L. Such alternative embodiments that implement the just-described opposite interaction require the user to move and/or re-focus their eyes in order to track screen 301 on device 300. Moving the device 300 with the left hand 303L has another disadvantage, namely the camera 310 is likely to be tilted during such movement which results in a large movement of image 309 on screen 301, typically larger than the dimensions of device 300.
Several embodiments evaluate the first and second predetermined conditions described above based on distances and/or movements of object 302 relative to a real world scene 390 (which includes a coffee cup 391). In such embodiments, when object 302 is kept stationary or steady relative to scene 390, the first and second predetermined conditions are not satisfied simply by manual movement of device 300 through distance dX (relative to object 302 and scene 390, both of which are stationary or steady). Instead, such embodiments are designed with the assumption that it is device 300 that is being kept stationary or steady, while object 302 is moved relative to scene 390.
Device 300 remains “steady” (as this term is used in this detailed description) even when device 300 is not strictly stationary. Specifically, device 300 remains “steady” even when device 300 is moved (relative to scene 390) through distances in the real world that are too small to be perceptible by an eye of a human, such as involuntary movements that may be inherent in a hand of the human. Therefore, although camera 310 may move around a little in the real world due to involuntary movement of a hand of the human intending to hold device 300 stationary, any such movement of camera 310 relative to scene 390 is smaller (e.g. three times, five times or even ten times (i.e. an order of magnitude) smaller) than movement through distance dX of object 302 relative to scene 390 that satisfies the second predetermined condition. Hence, some embodiments filter out involuntary movements by use of a threshold in the second predetermined condition.
Several embodiments are designed with no assumption as to device 300 being kept stationary (or steady, depending on the embodiment) relative to scene 390. Instead, device 300 of such embodiments measures a first relative motion between camera 310 and object 302 and also measures a second relative motion between camera 310 and scene 390, and then computes a difference between these two relative motions to obtain a third relative motion between object 302 and scene 390. Device 300 of the just-described embodiments then uses the third relative motion to evaluate a first predetermined condition and/or a second predetermined condition of the type described above.
In an example of the type described above in reference to
In some aspects of described embodiments, repeated movements by user 303, e.g. every hour to view emails received in the last hour, result in a form of training of the user's right hand 303R, so that user 303 automatically performs a specific movement (like a gesture) to issue a specific command to device 300, yielding faster performance than any prior art menu selection techniques known to the current inventor.
Accordingly, the just-described movement of object 302 to perform a menu selection in device 300 is a new interaction technique that is also referred to herein as target based menus. Target based menus in accordance with several embodiments use movements of object 302 by a user's hand to facilitate complex selections of apps and menus arranged as a layered pie in three dimensions, and successively displayed on screen 301 as described below in reference to
In embodiments of the type shown in
To support such a layered pie of menus, a flow of acts illustrated in
In some embodiments, second menu 503 shown in
In several embodiments, multiple menus of such a layered pie are associated with (or included in) corresponding apps in mobile device 300. Accordingly, associations between multiple menus and their associated apps are predetermined and stored in mobile device 300 ahead of time, for use by a processor 306 in acts 206-209 depending on the Z axis distance. In this way, user 303 is able to select a menu area very quickly from a hierarchy of menus arranged as a layered pie without using any button on device 300 or touching the screen of device 300, e.g. just by performing a gesture like movement with a predetermined object in 3-D space in the vicinity of device 300.
Although in some embodiments a single predetermined object 302 is associated with multiple menus 304 and 503, in other embodiments different menus (or menu hierarchies) are associated with different predetermined objects, by associations that are predetermined and stored in mobile device 300. Specifically, in some embodiments, when object 302 is detected by mobile device 300 in evaluating the first predetermined condition, an identity 381 of object 302 is used with an association 371 in memory 319 to identify a corresponding menu 304 in app 320 (
Hence, a menu selection technique of the type described herein in reference to
Moreover, as noted above, due to the wide field of view of the optical lens 311 in camera 310 of mobile device 300, even small movements of a predetermined object are magnified into large movements of the corresponding on-screen image. This magnification effect enables user 303 to operate very quickly and perform menu-based tasks via very small movements (similar to movements of a mouse on a mouse pad of a desktop computer). The new technique leads to a speed up in the process. Also, specific selections are related to specific a combined movement (for example up-right).
Several acts of the type described herein are performed by one or more processors 306 included in mobile device 300 (
Mobile device 300 includes a camera 310 (
Mobile device 300 may also include a disk (or SD card) 1008 to store data and/or software for use by processor(s) 306. Mobile device 300 may further include a wireless transmitter and receiver in transceiver 1010 and/or any other communication interfaces 1009. It should be understood that mobile device 300 may be any portable electronic device such as a cellular or other wireless communication device, personal communication system (PCS) device, personal navigation device (PND), Personal Information Manager (PIM), Personal Digital Assistant (PDA), laptop, camera, smartphone, or other suitable mobile platform that is capable of creating an augmented reality (AR) environment.
In an Augmented Reality environment there might be different interaction metaphors used. Tangible interaction allows a user 303 to reach into scene 390 that includes various real world objects, such a cup 391 of steaming coffee and a business card being used as predetermined object 302 (
Menu areas that are displayed on screen 301 and selected by real world movements in scene 390 as described herein can have a broad range of usage patterns. Specifically, such menus areas can be used in many cases and applications similar to menu areas on touch screens that otherwise require embodied interaction. Moreover, such menu areas can be used in an AR setting even when there is no touch screen available on mobile phones. Also, use of menu areas as described herein allows a user to select between different tools very easily and also to use the UI of the mobile device in the normal manner, to specify specific commands already known to the user. This leads to much faster manipulation times. Accordingly, menus as described herein cover a broad range of activities, so it is possible to use menus as the only interaction technique for a whole application (or even for many different applications). This means once a user has learned to select items in a menu by tangible interaction with augmented reality (AR) applications as described herein, the user will not need to learn any other tool to issue commands to AR applications.
A mobile device 300 of the type described above may include other position determination methods such as object recognition using “computer vision” techniques. The mobile device 300 may also include means for remotely controlling a real world object that is being used as predetermined object 302 which may be a toy, in response to the user input via menu selection, e.g. by use of transmitter in transceiver 1010, which may be an IR or RF transmitter or a wireless a transmitter enabled to transmit one or more signals over one or more types of wireless communication networks such as the Internet, WiFi, cellular wireless network or other network. The mobile device 300 may further include, in a user interface, a microphone and a speaker (not labeled). Of course, mobile device 300 may include other elements unrelated to the present disclosure, such as a read-only-memory 1007 which may be used to store firmware for use by processor 306.
Although the present invention is illustrated in connection with specific embodiments for instructional purposes, the present invention is not limited thereto. Hence, although item 300 shown in
Various adaptations and modifications may be made without departing from the scope of the invention. Therefore, the spirit and scope of the appended claims should not be limited to the foregoing description. It is to be understood that several other aspects of the invention will become readily apparent to those skilled in the art from the description herein, wherein it is shown and described various aspects by way of illustration. The drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.