Patent Application
20140160089
The present invention relates to an interactive input system and to an input tool therefor.
Interactive input systems that allow users to input ink into an application program using an active pointer (e.g. a pointer that emits light, sound or other signal), a passive pointer (e.g. a finger, cylinder or other object) or other suitable input device such as for example, a mouse or trackball, are well known. These interactive input systems include but are not limited to: touch systems comprising touch panels employing analog resistive or machine vision technology to register pointer input such as those disclosed in U.S. Pat. Nos. 5,448,263; 6,141,000; 6,337,681; 6,747,636; 6,803,906; 7,232,986; 7,236,162; and 7,274,356 and in U.S. Patent Application Publication No. 2004/0179001 assigned to SMART Technologies ULC of Calgary, Alberta, Canada, assignee of the subject application, the disclosures of which are incorporated by reference in their entireties; touch systems comprising touch panels employing electromagnetic, capacitive, acoustic or other technologies to register pointer input; tablet and laptop personal computers (PCs); smartphones, personal digital assistants (PDAs) and other handheld devices; and other similar devices.
Above-incorporated U.S. Pat. No. 6,803,906 to Morrison et al. discloses a touch system that employs machine vision to detect pointer interaction with a touch surface on which a computer-generated image is presented. A rectangular bezel or frame surrounds the touch surface and supports digital imaging devices at its corners. The digital imaging devices have overlapping fields of view that encompass and look generally across the touch surface. The digital imaging devices acquire images looking across the touch surface from different vantages and generate image data. Image data acquired by the digital imaging devices is processed by on-board digital signal processors to determine if a pointer exists in the captured image data. When it is determined that a pointer exists in the captured image data, the digital signal processors convey pointer characteristic data to a master controller, which in turn processes the pointer characteristic data to determine the location of the pointer in (x,y) coordinates relative to the touch surface using triangulation. The pointer coordinates are conveyed to a computer executing one or more application programs. The computer uses the pointer coordinates to update the computer-generated image that is presented on the touch surface. Pointer contacts on the touch surface can therefore be recorded as writing or drawing or used to control execution of application programs executed by the computer.
U.S. Pat. No. 7,532,206 to Morrison et al. discloses a touch system that comprises a touch surface and at least one camera acquiring images of the touch surface. A pointer contact data generator generates pointer position data in response to pointer contact with the touch surface, the pointer position data representing where on the touch surface pointer contact is made. A processor communicates with the at least one camera and the pointer contact data generator. The processor analyzes acquired images to determine the type of pointer used to contact the touch surface, and processes the pointer position data in accordance with the determined type of pointer. In one embodiment, the processor distinguishes between pointer tip touch surface contacts, pointer backend touch surface contacts and finger touch surface contacts. A writing function is invoked in response to pointer tip touch surface contacts. An erase function is invoked in response to pointer backend touch surface contacts. Mouse events are generated in response to finger touch surface contacts.
U.S. Pat. No. 7,202,860 to Ogawa discloses a coordinate input device that includes a pair of cameras positioned in upper left and upper right positions of a display screen of a monitor lying close to a plane extending from the display screen of the monitor, and views both a side face of an object in contact with a position on the display screen and a predetermined desk-top coordinate detection area to capture an image of the object within the field of view. The coordinate input device also includes a control circuit which calculates a coordinate value of a pointing tool, pointing to a position within a coordinate detection field, based on video signals output from the pair of cameras, and transfers the coordinate value to a program of a computer.
Although the above-described interactive input systems are satisfactory, improvements are desired. It is therefore an object at least to provide a novel interactive input system, and a novel input tool therefor.
Accordingly, in one aspect there is provided an input tool for use with an interactive input system, the input tool comprising a body housing processing circuitry storing input tool operating mode data representing operating modes of said input tool; and at least one display on the body and communicating with said processing circuitry, said display being responsive to said processing circuitry to present a selected operating mode of said input tool.
In one embodiment, the processing circuitry is responsive to user input to select one of the input tool operating modes for presentation on the display. The input tool operating modes may be grouped into categories with the operating modes of a user selected category being selectable for presentation on the display.
In one embodiment, at least one manually actuable element is provided on the housing. The processing circuitry is responsive to actuation of the at least one element to enable user selection of an input tool operating mode.
In one embodiment, the processing circuitry wirelessly broadcasts the selected input tool operating mode. In one form, the selected input tool operating mode may be used to modulate a signal broadcast by the processing circuitry. The display in one form is configured to display textual and graphical information and may present a graphical image icon of the selected input tool operating mode. The graphical image icon may generally correspond to an appearance of digital ink associated with the selected input tool operating mode.
According to another aspect there is provided an input tool for use with an interactive input system, the input tool comprising a pair of arms rotatably connected together by a hinge; a sensor providing output representing the angle formed between said arms; and processing circuitry communicating with said sensor and outputting a signal comprising angle information.
In one embodiment, the sensor comprises a potentiometer. The processing circuitry wirelessly broadcasts the signal and the angle information may be used to modulate the broadcast signal.
According to another aspect there is provided an input tool for use with an interactive input system, the input tool comprising an elongate body; and a deformable nib at one end of said body configured to contact an interactive surface.
In one embodiment, the deformable nib is generally conical and wherein the shape of the deformable nib becomes less conical and more cylindrical as pressure applied to the interactive surface by the deformable nib increases. The deformable nib may be fabricated of visco-elastic polyurethane foam that is coated with a polytetrafluoroethylene-based material.
According to another aspect there is provided an interactive input system comprising at least one imaging assembly capturing image frames of a region of interest; and processing structure communicating with said at least one imaging assembly and processing said captured image frames to determine the shape of an input tool tip appearing therein; and generate tip pressure data based on the determined shape of said input tool tip.
Embodiments will now be described more fully with reference to the accompanying drawings in which:
Turning now to
The interactive board 22 employs machine vision to detect one or more pointers brought into a region of interest in proximity with the interactive surface 24. The interactive board 22 communicates with a general purpose computing device 28 executing one or more application programs via a universal serial bus (USB) cable 30 or other suitable wired or wireless connection. General purpose computing device 28 processes the output of the interactive board 22 and adjusts image data that is output to the projector, if required, so that the image presented on the interactive surface 24 reflects pointer activity. In this manner, the interactive board 22, general purpose computing device 28 and projector allow pointer activity proximate to the interactive surface 24 to be recorded as writing or drawing or used to control execution of one or more application programs executed by the general purpose computing device 28.
The bezel 26 in this embodiment is mechanically fastened to the interactive surface 24 and comprises four bezel segments 40, 42, 44, 46. Bezel segments 40 and 42 extend along opposite side edges of the interactive surface 24 while bezel segments 44 and 46 extend along the top and bottom edges of the interactive surface 24 respectively. In this embodiment, the inwardly facing surface of each bezel segment 40, 42, 44 and 46 comprises a single, longitudinally extending strip or band of retro-reflective material. To take best advantage of the properties of the retro-reflective material, the bezel segments 40, 42, 44 and 46 are oriented so that their inwardly facing surfaces extend in a plane generally normal to the plane of the interactive surface 24.
A tool tray 48 is affixed to the interactive board 22 adjacent the bezel segment 46 using suitable fasteners such as for example, screws, clips, adhesive etc. As can be seen, the tool tray 48 comprises a housing 48a having an upper surface 48b configured to define a plurality of receptacles or slots 48c. The receptacles 48c are sized to receive input tools such as pen tools and an eraser tool that can be used to interact with the interactive surface 24. Control buttons 48d are provided on the upper surface 48b to enable a user to control operation of the interactive input system 20. One end of the tool tray 48 is configured to receive a detachable tool tray accessory module 48e while the opposite end of the tool tray 48 is configured to receive a detachable communications module 48f for remote device communications. The housing 48a accommodates a master controller 50 (see
Imaging assemblies 60 are accommodated by the bezel 26, with each imaging assembly 60 being positioned adjacent a different corner of the bezel. The imaging assemblies 60 are oriented so that their fields of view overlap and look generally across the entire interactive surface 24. In this manner, any pointer such as for example a user's finger, a cylinder or other suitable object, or a pen or eraser tool lifted from a receptacle 48c of the tool tray 48, that is brought into proximity with the interactive surface 24 appears in the fields of view of the imaging assemblies 60. A power adapter 62 provides the necessary operating power to the interactive board 22 when connected to a conventional AC mains power supply.
Turning now to
A digital signal processor (DSP) 72 such as that manufactured by Analog Devices under part number ADSP-BF522 Blackfin or other suitable processing device, communicates with the image sensor 70 over an image data bus 74 via a parallel port interface (PPI). A serial peripheral interface (SPI) flash memory 74 is connected to the DSP 72 via an SPI port and stores the firmware required for image assembly operation. Depending on the size of captured image frames as well as the processing requirements of the DSP 72, the imaging assembly 60 may optionally comprise synchronous dynamic random access memory (SDRAM) 76 to store additional temporary data as shown by the dotted lines. The image sensor 70 also communicates with the DSP 72 via a a two-wire interface (TWI) and a timer (TMR) interface. The control registers of the image sensor 70 are written from the DSP 72 via the TWI in order to configure parameters of the image sensor 70 such as the integration period for the image sensor 70.
In this embodiment, the image sensor 70 operates in snapshot mode. In the snapshot mode, the image sensor 70, in response to an external trigger signal received from the DSP 72 via the TMR interface that has a duration set by a timer on the DSP 72, enters an integration period during which an image frame is captured. Following the integration period after the generation of the trigger signal by the DSP 72 has ended, the image sensor 70 enters a readout period during which time the captured image frame is available. With the image sensor in the readout period, the DSP 72 reads the image frame data acquired by the image sensor 70 over the image data bus 74 via the PPI. The frame rate of the image sensor 70 in this embodiment is between about 900 and about 960 frames per second. The DSP 72 in turn processes image frames received from the image sensor 72 and provides pointer information to the master controller 50 at a reduced rate of approximately 120 points/sec. Those of skill in the art will however appreciate that other frame rates may be employed depending on the desired accuracy of pointer tracking and whether multi-touch and/or active pointer identification is employed.
Strobe circuits 80 communicate with the DSP 72 via the TWI and via a general purpose input/output (GPIO) interface. The strobe circuits 80 also communicate with the image sensor 70 and receive power provided on power line 82 via the power adapter 52. Each strobe circuit 80 drives a respective illumination source in the form of a plurality of infrared (IR) light sources such as IR light emitting diodes (LEDs) 84 that provide infrared backlighting over the interactive surface 24 for the imaging assembly 60 during image capture. Further specifics of the strobe circuits are described in U.S. Patent Application Publication No. 2011/0169727 to Akitt filed on Feb. 19, 2010 and entitled “Interactive Input System Therefor”.
The DSP 72 also communicates with an RS-422 transceiver 86 via a serial port (SPORT) and a non-maskable interrupt (NMI) port. The transceiver 86 communicates with the master controller 50 over a differential synchronous signal (DSS) communication link 88 and a synch line 90. Power for the components of the imaging assembly 60 is provided on power line 92 by the power adapter 52. DSP 72 may also optionally be connected to a USB connector 94 via a USB port as indicated by the dotted lines. The USB connector 94 can be used to connect the imaging assembly 60 to diagnostic equipment.
The image sensor 70 and its associated lens as well as the IR LEDs 84 are mounted on a housing assembly 100 that is best illustrated in
Turning now to
As will be appreciated, the architectures of the imaging assemblies 60 and the master controller 50 are similar. By employing a similar architecture for both the imaging assemblies 60 and the master controller 50, the same circuit board assembly and common components may be used for both thus reducing the part count and cost of the interactive input system. Differing components are added to the circuit board assemblies during manufacture dependent upon whether the circuit board assembly is intending for use in an imaging assembly 60 or in the master controller 50. For example, the master controller 50 may require a SRAM 76 whereas the imaging assembly 60 may not.
The general purpose computing device 28 in this embodiment is a personal computer or other suitable processing device comprising, for example, a processing unit, system memory (volatile and/or non-volatile memory), other non-removable or removable memory (eg. a hard disk drive, RAM, ROM, EEPROM, CD-ROM, DVD, flash memory, etc.) and a system bus coupling the various computing device components to the processing unit. The general purpose computing device 28 may also comprise a network connection to access shared or remote drives, one or more networked computers, or other networked devices.
During operation, the DSP 200 of the master controller 50 outputs synchronization signals that are applied to the synch line 90 via the transceiver 208. Each synchronization signal applied to the synch line 90 is received by the DSP 72 of each imaging assembly 60 via transceiver 86 and triggers a non-maskable interrupt (NMI) on the DSP 72. In response to the non-maskable interrupt triggered by the synchronization signal, the DSP 72 of each imaging assembly 60 ensures that its local timers are within system tolerances and if not, corrects its local timers to match the master controller 50. Using one local timer, the DSP 72 initiates a pulse sequence via the snapshot line that is used to condition the image sensor to the snapshot mode and to control the integration period and frame rate of the image sensor 70 in the snapshot mode. The DSP 72 also initiates a second local timer that is used to provide output on the LED control line 174 so that the IR LEDs 84 are properly powered during the image frame capture cycle.
In response to the pulse sequence output on the snapshot line, the image sensor 70 of each imaging assembly 60 acquires image frames at the desired image frame rate. In this manner, image frames captured by the image sensor 70 of each imaging assembly 60 can be referenced to the same point of time allowing the position of pointers brought into the fields of view of the image sensors 70 to be accurately triangulated. Each imaging assembly 60 has its own local oscillator (not shown) and synchronization signals are distributed so that a lower frequency synchronization signal (e.g. the point rate, 120 Hz) for each imaging assembly is used to keep image frame capture synchronized. By distributing the synchronization signals for the imaging assemblies 60 rather than transmitting a fast clock signal to each imaging assembly from a central location, electromagnetic interference is reduced.
During image frame capture, the DSP 72 of each imaging assembly 60 also provides output to the strobe circuits 80 to control the switching of the IR LEDs 84. When the IR LEDs 84 are on, the IR LEDs flood the region of interest over the interactive surface 24 with infrared illumination. Infrared illumination that impinges on the retro-reflective bands of bezel segments 40, 42, 44 and 46 and on the retro-reflective labels 118 of the housing assemblies 100 is returned to the imaging assemblies 60. As a result, in the absence of a pointer, the image sensor 70 of each imaging assembly 60 sees a bright band having a substantially even intensity over its length together with any ambient light artifacts. When a pointer is brought into proximity with the interactive surface 24, the pointer occludes infrared illumination reflected by the retro-reflective bands of bezel segments 40, 42, 44 and 46 and/or the retro-reflective labels 118. As a result, the image sensor 70 of each imaging assembly 60 sees a dark region that interrupts the bright band in captured image frames. The reflections of the illuminated retro-reflective bands of bezel segments 40, 42, 44 and 46 and the illuminated retro-reflective labels 118 appearing on the interactive surface 24 are also visible to the image sensor 70.
The sequence of image frames captured by the image sensor 70 of each imaging assembly 60 is processed by the associated DSP 72 to remove ambient light artifacts and to identify each pointer in each image frame. The DSP 72 of each imaging assembly 60 in turn conveys the pointer data to the DSP 200 of the master controller 50. The DSP 200 uses the pointer data received from the DSPs 72 to calculate the position of each pointer relative to the interactive surface 24 in (x,y) coordinates using well known triangulation as described in above-incorporated U.S. Pat. No. 6,803,906 to Morrison. This pointer coordinate data together with pen tool operating mode information received by the DSP 200 from one or more pen tools via the external antenna 136 and wireless receiver 138, if any, are conveyed to the general purpose computing device 28 allowing the image data presented on the interactive surface 24 to be updated.
Turning now to
The interior of the body 300 houses processing circuitry mounted on a printed circuit board. In this embodiment, the processing circuitry comprises a controller 312 that communicates with a wireless unit 316. Wireless unit 316 is conditioned by the controller 312 to broadcast a modulated signal via a wireless transmitter 318, such as for example, a radio frequency (RF) antennae or one or more illumination sources, such as IR LEDs, when the pen tool P is powered on through actuation of button 310. The signal broadcast by the pen tool P is modulated by operating mode information that identifies the currently selected operating mode of the pen tool P. The controller 312 also communicates with memory 314 that stores information associated with the pen tool operating modes. In this embodiment, the memory 314 stores, for each selectable pen tool operating mode, its operating mode category, a graphical image icon of the pen tool operating mode, and the operating mode information that is used to modulate the signal broadcast by the pen tool P. The graphical image icon generally corresponds to an appearance of digital ink associated with the pen tool operating mode. A battery 320 supplies power to the processing circuitry.
With the pen tool P powered on, when a user depresses button 306, the controller 312 cycles through the operating mode categories stored in the memory 314, and in turn displays the name of each operating mode category on the display 304 in succession. In this manner, the user can press the button 306 until the desired pen tool operating mode category appears on the display 304 thereby to select that pen tool operating mode category. Once a pen tool operating mode category has been selected, when the user depresses button 308, the controller 312 cycles through the pen tool operating modes of the selected operating mode category, and in turn displays a graphical image representation of those pen tool operating modes on the display 304 in succession.
Once a threshold time period has passed after button 306 and/or button 308 has been depressed resulting in an operating mode category and a pen tool operating mode thereof being selected, in this embodiment one (1) second, the controller 312 conditions the wireless unit 316 to continuously broadcast a signal modulated with the operating mode information associated with the selected pen tool operating mode. Of course, a different threshold time period may be employed or the wireless unit may be configured to broadcast automatically a modulated signal upon powering up of the pen tool P. In this latter case, the signal may be broadcast with the last selected pen tool operating mode or a default pen tool operating mode being used to modulate the broadcast signal.
The DSP 200 stores a modulated signal-to-pen tool operating mode mapping table in the memory 202. When the pen tool P is brought into proximity with the interactive surface 24 and a broadcast modulated signal is received by the DSP 200 via the antenna 136 and wireless receiver 138, the DSP 200 compares the received modulated signal to the mapping table to determine the pen tool operating mode. The DSP 200 in turn uses this information to assign mode information to the generated pointer coordinates, and conveys the mode information along with the pointer coordinates to the general purpose computing device 28 so that the pointer coordinates are processed by the general purpose computing device 28 in the desired manner. The general purpose computing device 28 treats the pointer coordinates as either marker (i.e. ink) events, eraser events, mouse events, or other events, in accordance with the mode information.
Although in the embodiment described above, the pen tool P comprises a color LCD that is capable of displaying textual and graphical information in a range of colors, in other embodiments, the pen tool may alternatively comprise another type of display. For example, in other embodiments, the pen tool may comprise a monochromatic LCD, a single-line or multi-line display that is capable of displaying only alphanumeric information, and/or other types of displays that are capable of displaying alphanumeric information and/or graphical information.
Although in the embodiment described above, the pen tool comprises buttons which may be depressed for selecting the operating mode category and the operating mode of the pen tool, in other embodiments, the pen tool may comprise other configurations. For example, the pen tool may comprise one or more switches capable of being toggled through multiple positions, one or more rotating switches, one or more scroll wheels, and/or one or more pressure or orientation sensitive switches etc., actuable to allow the operating mode category and operating mode of the pen tool P to be selected. In other embodiments, the pen tool may further comprise a microphone and the controller may be configured to execute voice recognition software to enable the operating mode category and pen tool operating mode to be selected by the user by voice command input into the microphone. In still other embodiments, the operating mode category and operating mode of the pen tool may also be selected through haptic commands, such as through pointer input on the interactive surface 24. In a related embodiment, the display of the pen tool may be touch sensitive, and may be configured to receive touch input for selection of the operating mode category and operating mode of the pen tool.
The compass 440 comprises processing circuitry mounted on a printed circuit board (not shown) that is housed within the first arm 442. The processing circuitry comprises a controller 452 that communicates with a sensor in the form of a potentiometer 454 housed within the hinge 446. The potentiometer 454 is configured to provide output proportional to the angle θ formed by the arms 442 and 444 to the controller 452. The controller 452 also communicates with a wireless unit 456. Wireless unit 456 is conditioned by the controller 452 to broadcast a modulated signal via wireless transmitter 458 when the compass 440 is powered on via power on/off switch 460 located on arm 442 and battery 462. The broadcast signal is modulated by angle information that identifies the relative orientation of the arms 442 and 444.
In this embodiment, the DSP 200 stores a modulated signal-to-compass operating mode mapping table in the memory 202. When the compass 440 is brought into proximity with the interactive surface 24 and a broadcast modulated signal is received by the DSP 200 via the antenna 136 and wireless receiver 138, the DSP 200 compares the received modulated signal to the mapping table to determine the compass operating mode. The DSP 200 in turn uses this information to assign mode information to the generated pointer coordinates and conveys the mode information along with the pointer coordinates to the general purpose computing device 28 so that the pointer coordinates are processed by the general purpose computing device 28 in the desired manner.
In this embodiment, when the compass 440 is fully closed and the angle θ between first and second arms 442 and 444 is substantially equal to zero (0) degrees, the DSP 200 assigns geometric correction mode information to the generated pointer coordinates. In this mode, the general purpose computing device 28 geometrically corrects digital ink displayed at locations corresponding to the pointer coordinates. The geometrical corrections may comprise, for example, straightening lines that are not straight, and rendering non-true geometric objects (e.g. triangles, rectangles, pentagons, etc.) true. When the compass 440 is fully open and the angle θ between first and second arms 442 and 444 is equal to approximately 180 degrees, the DSP 200 assigns orthogonal ink mode information to the generated pointer coordinates. In this mode, the general purpose computing device 28 generates digital ink in the form of either vertical or horizontal lines, only, at the generated pointer coordinates. When the compass 440 is partially open and the angle θ between first and second arms 442 and 444 is between zero and 180 degrees, the DSP 200 assigns compass ink mode information to the generated pointer coordinates. In this mode, the general purpose computing device 28 generates digital ink in the form of generally perfect arcs or circular lines at the generated pointer coordinates.
If desired, a display may be provided on one or both of the arms for displaying the current compass operating mode. Similar to the previous embodiment, the display may be a colour or monochromatic LCD, a single or multi-line display or other suitable display.
Although specific input tool operating modes and operating mode categories have been described, those of skill in the art will appreciate that many other input tool operating modes and categories may be assigned.
Although in embodiments described above, the DSP 200 is shown as comprising an antenna and a wireless receiver for receiving the modulated signal output by the input tool, in other embodiments, each imaging assembly may alternatively be provided with an antenna and a wireless receiver for receiving the modulated signal output by the input tool. In other embodiments, the input tool may alternatively be tethered to the interactive board or to the DSP of the master controller for allowing the modulated signal output by the input tool to be conveyed by a wired connection.
In other embodiments, the input tool may alternatively comprise one or more tip switch assemblies, each of which is actuable when brought into contact with the interactive surface. In these embodiments, the controller conditions the wireless unit and transmitter to broadcast the modulated signal upon actuation of a tip switch assembly.
In other embodiments, the modulated signal that is broadcast by the input tool may also comprise a unique identifier of the input tool that allows the input tool to be distinguished from other input tools and/or other pointers by the interactive input system.
It will also be appreciated that other forms of input tools may be used with the interactive input system 20. For example,
During operation, the DSP 200 uses the pointer tip shape, received from the DSPs 72 with the pointer data, to determine the tip pressure applied to the interactive surface 24 by the stylus 540. The calculated pointer coordinates and the tip pressure are then conveyed by the DSP 200 to the general purpose computing device 28 via the USB cable 30. The general purpose computing device 28 in turn processes the received pointer coordinates and tip pressure data, and updates the image output provided to the display unit, if required, so that the image presented on the interactive surface 24 reflects the pointer activity.
Although in the embodiments described above, the imaging assemblies 60 are described as being positioned adjacent corners of the bezel, those of skill in the art will appreciate that the imaging assemblies may be placed at different locations relative to the bezel.
Those of skill in the art will also appreciate that, although the operation of the interactive input system 20 has been generally described with reference to a single input tool being positioned in proximity with the interactive surface, the interactive input system 20 is capable of detecting multiple input tools that are positioned in proximity with the interactive surface.
Although preferred embodiments have been described, those of skill in the art will appreciate that variations and modifications may be made with departing from the scope thereof as defined by the appended claims.
