Pointing device and method for item location and/or selection assistance

Abstract

The invention describes a pointing device (1) and a methode for item location and/or selection assistance, which method comprises visually presenting a number of items (M1, M2, M3, M4) in a visual presentation (VIP, VP′, aiming a pointing device (1) comprising a camera (2) and a directable source (12) of a concentrated beam of light (L) at the visual presentation (VP, VP′ of the items (M1, M2, M3, M4), generating image data (3) of a target area (A) at which the pointing device (1) is aimed, analysing the image data (3) in order to locate a specific point within the target area (A), generating control signals (9) for controlling the directing arrangement (4), and directing the concentrated beam of light (L) so that the point (P1) coincides with the specific point in the target area (A). The invention describes an interacting device (13) for interacting with a pointing device (1) to carry out I:he method for item location and/or selection assistance. Furthermore, the invention describes a system (14) for item location and/or selection assistance.

Description

Other objects and features of the present invention will become apparent from the following detailed descriptions considered in conjunction with the accompanying drawing. It is to be understood, however, that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the invention.

FIG. 1 is a schematic diagram of a pointing device and an interacting device in accordance with an embodiment of the present invention;

FIG. 2 is a schematic diagram of a pointing device in accordance with an embodiment of the present invention;

FIG. 3 is a schematic diagram of a visual presentation of a collection of items and a target area image of the visual presentation made by a pointing device, in accordance with an embodiment of the present invention;

FIG. 4 is a schematic diagram of a system for locating or selecting an item amongst a collection of items, in accordance with an embodiment of the present invention.

FIG. 5 is a schematic diagram showing a visual presentation and a corresponding target area image in accordance with an embodiment of the present invention.

In the drawings, like numbers refer to like objects throughout. The pointing device described is held and operated by a user, not shown in the drawings. The user can communicate by means of a suitable user interface, also not shown in the drawings, with the device for interacting with the pointing device.

FIG. 1 shows a pointing device 1 containing a camera 2 which generates images of the area in front of the pointing device 2 in the direction of pointing D. The pointing device 1 features an elongated form in this embodiment, so that the direction of pointing D lies along the longitudinal axis of the pointing device 1. The camera 2 is positioned towards the front of the pointing device 1 so that images are generated of the area in front of the pointing device 2 at which the user 8 is aiming. Image data 3 describing the images are transmitted by means of a communication interface 5 enclosed in the housing of the pointing device 1, and are transmitted in a wireless manner, e.g. Bluetooth, 802.11b or mobile telephony standards, to an interacting device 13. A receiving unit 10 in an interacting device 13, working together with the pointing device 1, receives the image data 3 and forwards them to an image analysis unit 6.

The received image data 3 are analysed in the image analysis unit 6 of the interacting device 13, where they are compared to other images or templates retrieved from an internal memory 20 or an external source 21, 22 by an accessing unit 19. Ideally, the accessing unit 19 has a number of interfaces allowing access to external data, for example the user might provide pre-defined templates stored on a memory medium 21 such as floppy disk, CD or DVD, or the accessing unit 19 might retrieve suitable template information from an external network such as the internet 22. The templates may also be configured by the user, for example in a training session in which the user specifies the correlation between specific areas on a template with particular items or functions.

The user in this case may be trying to locate an item, so that the image analysis unit 6 compares the image data 3 with the templates to determine whether the item sought is within the target area or not, and directs a control signal generator 8 to generate appropriate control signals 9, which are transmitted by a sending unit 11 of the interacting device 13 in a wireless manner to a communication interface 7 of the pointing device 1.

A laser light source 12, incorporated in the pointing device 1, emits a beam of laser light L in a direction not necessarily parallel to the direction of pointing D. The actual direction of the beam of laser light L is controlled by a directing arrangement 4 which applies the received control signals 9 to adjust the direction of pointing of the laser light source 12. The light point is directed in such a way that the user is eventually guided to the item being sought. In the figure, the directing arrangement 4 applies the control signals 9 to alter the position of the laser light source 12 accordingly, by means of, for example, a miniature motor. The beam of laser light L is thus aimed in the desired direction. In an alternative realisation, the directing arrangement 4 may comprise a number of small mirrors, whose position can be altered, and arranged in such a way that the mirrors deflect the beam of laser light L in the required direction. It is also feasible that a combination of miniature motor and mirrors might be used to control the direction of the beam of laser light L.

In another scenario, the pointing device 1 is being used to select an item, for example when training the interacting device to recognise and locate items. In this case, image data 3 is generated by aiming the pointing device at the item to be recognised, and is sent to the image analysis unit 6 to be analysed and processed in some way before being stored in a suitable format in the internal or external memories 20, 21.

In another application, the interacting device 13 features an interface 24 for communicating with an external device 25 such as a television, VCR, or any type of device with which a dialog might be initiated. Here, the interacting device 13 informs the external device 25 in some way of the user's actions. For example, the image analysis unit 6 determines, with the aid of templates for the options of this device 25, the area in the template at which the user is pointing, and sends this information to the external device 25, which interprets the information and send appropriate signals to the interacting device, where they are converted to control signals 9 for the directing arrangement 4 of the pointing device 1. In this way, the pointing device 1 together with the interacting device 13 can be used to assist the user in controlling or communicating with external devices 25.

FIG. 2 shows an embodiment of the pointing device 1 featuring its own image analysis unit 6′ and control signal generator 8′. This pointing device 1 can analyse image data 3 generated by its camera 2 to locally generate control signals 9 for the directing arrangement 4. Being able to perform the image processing locally means the pointing device 1 does not necessarily need to communicate with a separate interacting device 13 as described in FIG. 1. Since the quality of the image analysis might be limited by the physical dimensions of the pointing device 1, which will most likely be realised in a small and practical format, this “stand-alone” embodiment might suffice for situations in which the accuracy of the image analysis is not particularly important, or in situations where the pointing device 1 is unable to communicate with an interacting device. This embodiment may of course be simply an extension of FIG. 1, so that the pointing device 1 also avails of the communication interfaces 5, 7 described in FIG. 1, allowing it to operate in conjunction with an interacting device 13 such as a dialog system in addition to its stand-alone functionality. This embodiment might also features a local memory, not shown in the diagram, in which the pointing device 1 can store images generated by the camera 2.

FIG. 3 shows a visual presentation VP, in this case a number of actual objects M₁, M₂, M₃, M₄ on a shelf. A pointing device 1 is being aimed at a target area T of this visual presentation VP to select or locate one of the objects M₁, M₂, M₃, M₄.

If the user wants to locate an object, for example the item M₄, he might request an interacting device (not shown in the drawing) to assist him in locating it. Images 16 of the target area T are transmitted at intervals to the interacting system, where they are analysed to determine the area at which the pointing device 1 is aimed, and whether this area contains the item M₄ being sought. As long as this item M₄ cannot be detected in the image 16 of the target area T, the light source 12 of the pointing device 1 is directed by means of control signals so that the ensuing light point PL is moved in such a way as to indicate to the user the direction in which he must aim the pointing device 1 so that the item M₄ can ultimately be detected in the image 16 of the target area T, at which stage the light point PL is positioned over the desired item M₄ to show the user where it is. In the event that the system cannot locate an item because the item is missing or because the system is unable to understand the user's wishes, the light point PL might behave in a predefined manner e.g. by being turned on and off in a particular sequence, or by describing a predefined pattern. This would be of use, when, for example, the interacting device is unable to communicate with user by means of speech.

Should the user wish to select one of the items M₁, M₂, M₃, M₄ visible in the visual presentation VP, for example when training the interacting device to remember the locations of objects or to recognise their appearance, the user can aim the pointing device 1 at the visual presentation VP so that the object in question is indicated by the light point P_L. When the pointing device is being used in such a training mode, the light point P_L can maintain a fixed position relative to the centre of the target area A, given by P_T. The light point P_L might be directed at a fixed position at a point removed from the centre point P_T or it might coincide with the centre point P_T.

With the aid of the light point P_L, the user can select one of the items M₁, M₂, M₃, M₄ shown in the visual presentation VP. A camera in the pointing device generates an image of the target area T centred around an image centre point P_T. The light point P_L also appears in the target area image. In this example, the light point P_L appears at a very small distance away from the image centre point P_T, so that the user can use the light point P_L to accurately point out items to the interacting device, in this case the item M₃. The user then describes the object M₃ for the interacting device, for example by saying “This book is ‘Middlemarch’ by George Eliot”, so that the interacting device performs any necessary image processing before storing the information describing the item M₃ to memory.

FIG. 4 shows a pointing device 1, an interacting device 13 and a visual presentation VP giving a system 14 for item location and/or selection assistance.

The interacting device 13 is in this example might be incorporated in some kind of home dialog system, allowing the user to communicate with it by means of spoken commands. For example, the user has asked the interacting device 13 a question, such as “Where is my Dire Straits CD ‘Money for Nothing’?”. The user aims the pointing device 1 in the general direction of the shelves on which his CD collection is kept, and allows the interacting device 13, in conjunction with the pointing device 1, to show him where the requested CD is kept. The interacting device 13, which has been trained in a previous training session to remember the locations of all the CDs in the collection, now sends control signals to the directing arrangement of the pointing device 1 so that the light point P_L is directed at the requested CD. If the requested CD is located within the target area T, the light point comes to rest on this CD, or might be caused to describe a tight circle over the CD. However, if the CD is outside of the target area T, the control signals issued by the interacting device 13 cause the light point P_L to repeatedly move against the appropriate edge of the target area T, so that the user will realise that he must move the pointing device 1 in the indicated direction until the target area T includes the requested CD.

In this embodiment, the pointing device 1 also features a button 15. The button 15 can be pressed by the user, for example to confirm that he has made a selection and to record the image of the target area.

Alternatively or additionally, such a button 15 might be used to activate or deactivate displaying of a dynamic visual presentation VP′ on, for example, a television screen, so that items or options are only displayed on the screen when actually required by the user. Alternatively, the function of the button 15 or a different button on the pointing device 1 might be to activate or deactivate the light source 12 incorporated in the pointing device 1, to activate or deactivate the pointing device 1 itself, or to switch between “locate” and “select” modes of operation. The pointing device 1 might be activated by means of a motion sensor incorporated in the pointing device 1, so that the laser light source is activated when the user takes hold of the pointing device 1, and that the pointing device starts to send images of the target area to the interacting device as soon as it the pointing device is taken up or moved.

The pointing device 1 draws its power from one or more batteries, not shown in the figure. Depending on the consumption of the pointing device 1, it may be necessary to provide a cradle into which the pointing device 1 can be placed when not in use, to recharge the batteries.

The user will not always aim the pointing device at right angles to the visual presentation—it is more likely that the pointing device will be aimed at a more or less oblique angle to the visual presentation, since it is easier to wave the pointing device that it is to change one's own position. This is illustrated in FIG. 5, which shows a schematic representation of a target area image 16 generated by a pointing device, not shown in the diagram, which is aimed at the visual presentation VP′ from a distance and at an oblique angle, so that the scale and perspective of the items M₁, M₂, M₃ in the visual presentation VP′ appear distorted in the target area image 16. In the case shown in FIG. 5, the visual presentation VP′ is a television screen and the items M₁, M₂, M₃ from among which the user can choose at are menu items displayed on the screen.

Regardless of the angle of the pointing device 1 with respect to the visual presentation VP′, the target area image 16 is always centred around an target point P_T. The laser point P_L also appears in the target area image 16, and may be a distance removed from the target point P_T, or might coincide with the target point P_T. The image processing unit of the dialog system compares the target area image 16 with pre-defined templates to determine the item being pointed at the user, or to determine the location of the target point relative to the location of an item which the user is trying to locate.

To this end, the point of intersection P_T of the longitudinal axis of the pointing device 1 with the visual presentation VP′ is located. The point in the template corresponding to the point of intersection P_T can then be located. Computer vision algorithms using edge- and corner detection methods are applied to locate points in the target area image [(x_a, y_a), (x_b, y_b), (x_c, y_c)] which correspond to points in the template [(x_a′, y_a′), (x_b′, y_b′), (x_c′, y_c′)] of the visual presentation VP′.

Each point can be expressed as a vector e.g. the point (x_a, y_a) can be expressed as {right arrow over (v)}_a. As a next step, a transformation function T_λ is developed to map the target area image to the template:

$f\left(\lambda \right)=\sum _i{T_{\lambda }\left({\overrightarrow{v}}_i\right)-{\overrightarrow{v}}_i^{\prime }}^2$

where the vector {right arrow over (v)}_i represents the coordinate pair (x_i, y_i) in the target area image, and the vector {right arrow over (v)}_i′ represents the corresponding coordinate pair (x′_i, y′_i) in the template. The parameter set λ, comprising parameters for rotation and translation of the image yielding the most cost-effective solution to the function, can be applied to determine the position and orientation of the pointing device 1 with respect to the visual presentation VP. The computer vision algorithms make use of the fact that the camera 2 within the pointing device 1 is fixed and “looking” in the direction of the pointing gesture. The next step is to calculate the point of intersection of the longitudinal axis of the pointing device 1 in the direction of pointing D with the plane of the visual presentation VP. This point may be taken to be the centre of the target area image P_T. Once the coordinates of the point of intersection have been calculated, it is a simple matter to locate this point in the template of the visual presentation VP.

Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention. The pointing device can serve as the universal user interface device in the home or for navigation through business presentations. Outside of the home, it can be used in any environment where the user can be guided by means of the light point. In short, it can be beneficial wherever the user can express an intention by pointing, or wherever something can be actively pointed out to the user. Its small form factor and its convenient and intuitive usage can elevate such a simple pointing device to a powerful universal remote control or teaching tool. As an alternative to the pen shape, the pointing device could for example also be a personal digital assistant (PDA) with a built-in camera, or a mobile phone with a built-in camera. The pointing device might be combined with other traditional remote control features, e.g. with additional buttons for performing dedicated functions, or with other input modalities such as voice control.

For the sake of clarity, it is also to be understood that the use of “a” or “an” throughout this application does not exclude a plurality, and “comprising” does not exclude other steps or elements. A “unit” may comprise a number of blocks or devices, unless explicitly described as a single entity.

Claims

1. A pointing device, comprising a camera for generating image data of a target area in the direction in which the pointing device is aimed;a source of a concentrated beam of light for generating a light point within the target area; anda directing arrangement for directing the concentrated beam of light at any point in the target area .

2. A pointing device (1) according to claim 1, comprising a communication interface for transferring image data to an image analysis unit; anda communication interface for receiving from a control signal generation unit control signals for controlling the directing arrangement.

3. A pointing device according to claim 1, further comprising an image analysis unit for analysing image data; anda control signal generation unit for generating control signals for controlling the directing arrangement.

4. An interacting device or interacting with a pointing device according to claim 2, comprising a receiving unit for receiving image data from the pointing device;an image analysis unit for analysing the received image data;a control signal generation unit for generating control signals for controlling the directing arrangement of the pointing device;a sending unit for sending the control signals to the pointing device;

5. A system for item location and/or selection assistance comprising a pointing device according to claim 3, and a visual presentation of a number of items.

6. (canceled)

7. A method for item location and/or selection assistance, comprising: visually presenting a number of items in a visual presentation;aiming a pointing device comprising a camera and a directable source of a concentrated beam of light at the visual presentation of the items;generating image data of a target area at which the pointing device is aimed;analysing the image data in order to locate a specific point within the target area;generating control signals for controlling the directing arrangement; anddirecting the concentrated beam of light so that the light point coincides with the specific point in the target area.

8. A method according to claim 7, where a visual presentation of the items is presented in static form.

9. A method according to claim 7, where a visual presentation of the items is presented dynamically.

10. A method according to claim 7, wherein image data of the target area are analysed by comparing it to a predefined template of the target area and/or the visual presentation.

11. A method according to claim 7, where a located item is shown to the user by directing the light point at the located item.

12. A method according to claim 7, where a selected item is determined by locating a point in the template corresponding to a target point in the visual presentation at which a user has aimed the pointing device.

13. A method of according to claim 12, where the target point is determined by a method comprising the following steps: detecting distinctive points in the image data of the visual presentation; determining corresponding points in the template of the visual presentation;developing a transformation for mapping the points in the image data to the corresponding points in the template;using the transformation to determine the position and aspect of the pointing device relative to the visual presentation;locating the intersection point of a certain axis of the pointing device with the visual presentation.