Patent Application
20190041997
The present application relates to the field of computer device interfaces and, more specifically, to devices configured to allow a pointer device to interact with a computing device without an interface for that pointer device.
Certain handheld devices or tablet devices do not provide an interface for permitting a user to use a mouse to interact with the computing device, as the computing device is designed to receive input as gestures made on a touchscreen, and not the motions and clicks of a mouse translated to cursor movements on a screen.
These computing devices lack an interface for the pointer device for receiving input information from the pointer device. Such computing devices include Apple's iOS devices (e.g. iPad® and iPhone®). However, as tablet users may frequently utilize their computing device as a portable device to the likes of a laptop, it would be advantageous to have a means for allowing a mouse to interact with the computing device, as using a mouse to control the computing device may be privileged in a work setting over tactile input for performing certain tasks.
Moreover, providing for a means of connecting certain peripheral devices to a computing device without an interface for those peripheral devices may allow for the connecting of specialized peripherals, such as those used by users afflicted with a specific handicap who have difficulty using the computing device by performing touch-related gestures to control the device.
A device that would allow the communicating of these peripheral devices, such as a mouse, with a computing device with an interface for that peripheral device, would be advantageous.
Applicant has discovered that using an external relay device (or defined herein as “relay device”), connected to a pointer device, as a pointer interface for a computing device that, or else, does not have an interface for the pointer device, allows for the computing device to receive cursor movement information from the pointer device. The external relay device would be connected to the computing device and to the pointer device. This cursor movement information may be translated by the relay device into HID commands that may be understood by the OS of the computing device for generating the cursor on the computing device's screen, moving the cursor across the screen, and clicking with the cursor on the screen at a desired location, all as a function of input provided by the pointer device to the relay device.
A first broad aspect is a pointer relay device for a handheld or tablet computer having at least one data interface port and an operating system configured to interact with a user via a touch screen interface, a keyboard interface accepting HID commands and without an interface to an external pointer device. The pointer relay device has an interface for connecting to the data interface port. The pointer relay device has a pointer device interface configured to establish a connection with a pointer device and receive pointer device movement signals and pointer button signals from the pointer device via the connection. The pointer relay device has a controller configured to send commands using the data interface port to enable cursor movement control via HID commands, receive the pointer device movement signals and the pointer button signals from the pointer device interface, and send HID commands causing cursor movement and pointer selection corresponding to the pointer device movement signals and the pointer button signals.
In some embodiments, the interface for connecting to the data interface port may be a wired interface. The data interface port may be configured to accept external power for powering the handheld or tablet computer, and the pointer relay device may be further configured to provide power to the handheld or tablet computer via the external keyboard interface. The interface for connecting to the data interface port may be a wireless interface. The connection between the pointer device interface and the pointer device may be wired. The wired connection between the pointer device interface and the pointer device may be a USB bridge. The connection between the pointer device interface and the pointer device may be wireless. The wireless connection between the pointer device interface and the pointer device may be a Bluetooth connection.
In some embodiments, the pointer relay device may have a battery. The pointer relay device may have an authentication chip to authenticate the pointer device. The authentication chip may be comprised in the interface for connecting to the data interface port.
In some embodiments, the HID commands may be configured to be processed by an application program of the handheld or tablet computer that generates a cursor on the display of the handheld or tablet computer. In some embodiments, the controller may be further configured to send at least one keyboard command that causes the launch of the application program running on the computer that generates a cursor on the display.
In some embodiments, the pointer relay device may have an additional pointer device interface configured to establish a connection with an additional pointer device and receive pointer device movement signals and pointer button signals from the additional pointer device via the connection.
In some embodiments, the pointer device interface may be further configured to establish a second connection with an additional pointer device and receive pointer device movement signals and pointer button signals from the additional pointer device via the second connection.
In some embodiment, the interface for connecting to the data interface port may be further configured to be connected to a multiport adapter.
A second broad aspect is a method for performing cursor movement and pointer selection on a display of a handheld or tablet computer. The method includes sending HID commands to a data interface port of the handheld or tablet computer for causing cursor movement and pointer selection on the display corresponding to pointer device movement signals and pointer button signals of a pointer device.
In some embodiments, the method may involve, prior to the sending, authenticating the pointer device. The method may involve, prior to the sending, converting the pointer device movement signals and point button signals into corresponding HID commands.
In some embodiments, the HID commands that are sent may be configured to be processed by an application program of the handheld or tablet computer that generates a cursor on the display of the handheld or tablet computer.
In some embodiments, the method may further include sending at least one keyboard command to launch the application program that generates a cursor on the display. The method may further involve mirroring the display on an external screen connected to the handheld or tablet computer, wherein the pointer selection and cursor movement may be apparent on the external screen.
The invention will be better understood by way of the following detailed description of embodiments of the invention with reference to the appended drawings, in which:
Certain handheld devices or tablet devices do not allow an external pointer device (e.g. a mouse) to interact with the device as they do not have an interface for a mouse or the external pointer device. Such handheld devices or tablet devices are, for instance, those running an iOS (e.g. iPhone® and the iPad®). This is an inconvenience for users who desire to use a mouse to navigate and control their handheld or tablet device. The present application provides a relay device that serves as an interface between a peripheral device, such as an external pointer device, and the handheld or tablet device that does not have an interface for the external pointer device. The relay device then translates the input received by the peripheral device into HID commands that are sent to the handheld or tablet device to enable cursor movement and pointer button signals (the clicking or pressing of the buttons on the pointer device).
In the present application, a “computing device” is defined as a handheld computer or tablet computer with at least one data interface port, an operating system configured to interact with a user via a touch screen interface, and a keyboard interface accepting control key commands. The computing device, as defined herein, does not have an interface to an external pointer device (e.g. a mouse). Again, exemplary computing devices may be those running on iOS such as, for instance, the iPhone® and the iPad®.
Reference is now made to
The computing device has, for instance, computing device OS consumer control key module 32 that represents part of the computing device's 30 operating system that processes wireless keyboard commands and allows such commands as those to launch application programs or apps, enter text or make certain selections on the device. In the case of the Apple iPhone® or iPad®, keyboard commands can be used to perform actions that normally are associated with the device's touch screen actions or buttons, as for example, the swipe action to initiate unlocking a locked phone, the pressing of the home button, volume control, etc. Likewise, running a desired app can be implemented by using a keyboard command to initiate a search or find on the smartphone, and then sending keystrokes of the name of the app on the computing device 30 will cause the desired app 33 to be found, with another keystroke, such as ENTER. Keyboard commands may be used to activate certain application programs such as AssistiveTouch™.
A HID command is defined herein as a command sent by the relay device, processed by the OS of the computing device, that is related to input received by a human interface device (i.e. a peripheral device), such as a mouse, keyboard and joystick. Similarly to the keyboard commands, the HID commands may be processed by the computing device OS consumer control key module 32 that represents part of the computing device's 30. The input from the peripheral device is processed by the relay device to generate the HID commands, which may include HID reports related to the input from the peripheral device. The HID command, and the HID report(s) tied to the HID command, as defined herein, are not limited to a specific protocol or given standard such as the HID standard.
An example of a keyboard command that simulates a press on touch screen can be as follows:
It will be understood that the keyboard commands used to cause the smartphone to perform certain tasks, and as well the HID commands related to input received from a peripheral device, such as its unlocking or running a designated application, depends on the platform of the smartphone. The sequence of keyboard commands also depends upon the task to be carried out. Therefore, a skilled person will readily understand that a desired sequence of keyboard commands, or a desired HID command, for a specific platform may be determined using basic trial and observation, where the effect of receiving a specific sequence of keyboard commands (and HID commands) by the smartphone is monitored for the desired action.
In the example of
Exemplary input from a mouse (e.g. mouse 22a or mouse 22b) may be, for example, the movement of the pointer, or pointer button input such as, a click or press of the left button, a click or press of the right button, a “double-click” or “double-press” of the left button, or one click (or selection) or multiple clicks (or selections) of a button of the cursor. Exemplary input from a keyboard (e.g. keyboard 21a or keyboard 21b) may be a keystroke or a combination of keystrokes such as “ctrl+c” or “ctrl+z”. Such keystrokes may also include, for example, increasing or lowering the volume of the device, or the brightness of a screen.
The embodiment of
The relay device 15 may be used to process input from a peripheral device (e.g. mouse 22a) used by a local user to HID commands then processed by the local user's computing device 30. The mouse 22a, connected to a USB bridge 13a, transmits input, such as cursor movement information, or mouse click information, to the relay device 15 via its USB Bridge 13a.
The USB Bridge 13a (and/or the Bluetooth transceiver 13b) sends the input information from the mouse 22a (or mouse 22b) to the processor 11, such as movement signals and mouse button signals) that may use information on the configuration of the mouse 22a or 22b, or a set of translation settings for translating the input received from the mouse 22a or 22b into HID commands that are understood by the OS of the computing device 30, this information and/or configurations received by and/or stored in memory 14. The processor 11 then implements instructions, stored in memory 14, on converting the input into HID commands as a function of peripheral layout or configuration stored in memory 14. Processor 11 may be, e.g., a micro-processor. The HID commands may then be sent via a link, or wired link, between the relay device's 15 connector port 10 and the connector port 31 of the computing device 31. In some embodiments, as shown in
Once received, the computing device 30 carries out the desired action in correspondence with the HID commands, the tablet computer OS consumer control key module 32 receiving consumer control key code and/or descriptors, such as the HID relative pointer position reports, from the connector port 31 or the wireless interface 34. The computer OS consumer control key module 32 then sends the control commands or instructions to a target computer OS app 33 so they are carried out by the computer OS app 33. In the case where the peripheral device is a mouse, the target computer OS app 33 may be a program that allows for the appearance of the cursor, or for the motion of the cursor across the screen, as is described herein.
The computing device 30 may also send command settings relating to the keyboard 21a or 21b, or mouse 22a or 22b to the relay device 15 via, for example, a Bluetooth connection established between the computing device 30 and relay device 15. These command settings may then be stored in the non-volatile memory 14.
In some embodiments, the connector port 10 may also have an authentication chip 12 (e.g. MFi authentication chip for Apple™'s MFi Program), as is known in the art, for authenticating the peripheral device that is to communicate with the computing device 30 via the relay device 15. In other embodiments, the relay device 15 may have an authentication system for carrying out authentication protocols, such as one at least in part stored in memory 14 (e.g. in the case of software key authentication), in order to authenticate the peripheral device that is to be connected to the computing device 30 via the relay device 15.
In some embodiments, the HID commands sent by the relay device 15 may be specifically tailored to interact with the application program AssistiveTouch™ of an iOS device. AssistiveTouch™ is an application program for assisting a user in the controlling of the iOS device, such as in the performance of certain gestures (e.g. pinch, multi-finger swipe) and providing a shortcut for accessing certain features of the iOS device (e.g. the Control Center, Siri). Once the AssistiveTouch™ application program is activated as is described herein, AssistiveTouch™ may be configured in such a way that a cursor appears on the screen of the iOS device. Input from the peripheral device is converted into HID commands by the relay device 15, and these commands are sent to the iOS device by the relay device 15 and processed into commands of the cursor on the iOS device's screen, such as movements of the cursor, clicking on the screen or swiping across the screen. For example, where the peripheral device is a mouse 22a or 22b, the movement of the mouse 22a or 22b is converted into HID commands, the cursor replicating on the screen the movements of mouse 22a or 22b. In another example, when the peripheral device is a trackpad, the trackpad senses the movement and touch of, for example, a user's finger on its surface, and this input is sent to and converted into HID commands by the relay device 15, for operating the cursor appearing on the iOS device's screen, such as by moving, clicking or swiping of the cursor on the screen of the iOS device, these actions equivalent to the input received from the touchpad. The peripheral device, the relay device 15 and the Bluetooth connection between the peripheral device and the iOS device and/or between the relay device 15 and the iOS device may be MFi enabled (the MFi program is a licensing program run by Apple where hardware and software peripherals are enabled to run with Apple™ products, such as the iPhone and iPad). Furthermore, the relay device 15 may be part of the peripheral device. These illustrations are not limitative and are but examples of how the AssistiveTouch™ application program may be used in accordance with teachings of the present invention.
In one example, where the computing device 30 is an iOS device (where the OS is e.g. iOS 6 to iOS 10) the relay device 15 may transmit commands that are to be performed by the Switch Control™ application program of the iOS device. Switch Control™ is a program allowing a user to control the iOS device using certain input, such as a Bluetooth switch, tactile input registered by the device's screen, or a head motion registered by the device's camera. A first input allows the user to select a location on a screen, and a second subsequent input allows the user to choose an action (e.g. clicking on icon on a screen, swiping, etc.). The relay device 15 may transmit keyboard commands to activate Switch Control™ and then send keyboard commands to carry out certain actions using Switch Control™. For example, Switch Control™ may be useful for selecting an application on the screen, mimicking the tap of a finger. The Switch Control™ may be configured so that the relay device 15, following the reception of user input to select an application icon, would send keyboard commands, processed by the iOS device's iOS, to activate a first switch control of Switch Control™ to bring up the “HOME” page on the iOS device. Then, the relay device 15 may send keyboard commands to activate a second switch control to select the application icon. The second switch control may be, for instance, using Switch Control™'s item scanning feature, swoop scanning feature or manual scanning feature to identify the application icon with a sufficient time delay as required by the Switch Control™ application program, then do a timed selection of the application icon as permitted with Switch Control™. For example, once manual scanning is enabled, and using an external accessory for providing external input to the Switch Control™ enabled device, with knowledge of the layout of the home screen of the computing device, it is possible to use Switch Control™ to navigate through the applications on the home screen and perform, for instance, the selection of a desired application program.
The relay device 15 may be, for example, a standalone dongle. In other examples, the relay device may be hardware integrated into the peripheral (e.g. the keyboard, the mouse). For example, each peripheral device may have its own relay device communicating with the computing device 30 or multiple computing devices 30 (e.g. useful in a class setting).
Reference is now made to
A report of the relative position or pointer movement of the pointer device is sent to the iOS computing device via the relay device at step 301. The relay device translates the cursor movement information into HID commands relating to the relative position report of the pointer device. Then, the relay device transmits a keyboard command, “StartAssistiveTouch” Command, to the iOS computing device (e.g. via the connector port, or via a wireless connection) to initiate AssistiveTouch™ at step 302. Enabling AssistiveTouch™ prompts the appearance of a cursor on the touchscreen of the iOS computing device. A StartAssistiveTouchInformation command is then sent by the relay device to the iOS computing device (e.g. via the connector port, or via a wireless connection), the StartAssistiveTouchInformation command processed by the iOS and AssistiveTouch™ to generate a notification that AssistiveTouch™ is enabled at step 303. Once the notification that AssistiveTouch™ is enabled has been received by the relay device, where the notification is sent from the iOS computing device using, for instance, Apple™'s iAP2 protocol (where the relay device supports an iAP2 protocol), then a keyboard command is sent by the relay device to the iOS computing device to stop generating AssistiveTouch™ notifications at step 304 (e.g. via the connector port, or via a wireless connection).
As AssistiveTouch™ is enabled and a notification to this effect has been received, the relay device continues to receive cursor movement information from the pointer device, describing the motions and displacements of the pointer device at step 305. The cursor movement information may be received, for instance, via a USB bridge or a wireless Bluetooth interface of the relay device. The cursor movement information is then processed by the relay device into HID commands relating to HID pointer relative position reports at step 306, generated, for instance, from the pointer device configuration settings and processing instructions stored as software in the relay device's memory. An example of a mouse relative position report, that may be part of an HID command, is:
The HID commands relating to the HID pointer relative position reports are then transmitted using, for instance, a connector port (e.g. lightening port) via, for instance, a wired connection, to the connector port of the iOS computing device at step 307. In some examples, the HID commands relating to the HID pointer relative position reports may be transmitted to the iOS computing device via a wireless interface, such as a Bluetooth interface. However, using the bandwidth limited Bluetooth connection to transmit the HID commands may result in a jittery cursor movement across the screen of the iOS computing device. However, other versions of Bluetooth or any other wireless connection may provide sufficient responsiveness to achieve fluid cursor movement across the screen. In some examples, power may also be provided to the computing device via the connector port (e.g. lightning port) from the pointer relay device.
The computing device then processes and reads the HID commands relating to the HID pointer relative position reports and sends instructions to the AssistiveTouch™ Program as a function of the HID pointer relative position reports at step 308. The AssistiveTouch™ Program then carries out the instructions and moves the cursor on the screen as a function of the HID pointer relative position reports at step 309, the movement of the cursor corresponding to the movement of the pointer device.
Steps 305 and 309 are carried out periodically after given time intervals, in order to punctually provide information as the movement of the cursor that is translated by the relay device and then processed by the computing device's iOS, in order for the cursor's position to be regularly updated as a function of the pointer device's position, and also to allow for a smoother movement of the cursor on the screen of the iOS computing device. In one embodiment, steps 305 through 309 are carried out at 10 millisecond intervals where the cursor movement information is equally received and HID commands are provided on moving the cursor every 10 milliseconds (it will be appreciated that the 10 millisecond intervals are but an example time between different intervals and that the time intervals between each sequence of steps 305 and 309 may be other than (less or more) than 10 milliseconds without departing from the present teachings). Furthermore, a smoothing algorithm may be applied on cursor movement in order to provide for a smoother cursor movement when the pointer device undergoes a significant change in position. In one example, the smoothing algorithm may divide the relative position change of the pointer device, as provided in the cursor movement information, by a predefined factor.
Reference is now made to
Once AssistiveTouch™ is enabled, the cursor appearing on the screen of the computing device, and a notification to this effect has been received by the relay device, the relay device can also receive cursor click information from the pointer device, describing the click of a button of the cursor at step 311. The cursor click information may be received, for instance, via a USB bridge or a wireless Bluetooth interface of the relay device. The cursor click information is then processed by the relay device into HID commands relating to HID button click reports at step 312, generated, for instance, from the pointer device configuration settings and processing instructions stored as software in the relay device's memory.
The HID commands relating to the HID button click reports are then transmitted using, for instance, a connector port (e.g. lightening port) via, for instance, a wired connection, to the connector port of the iOS computing device at step 313. An exemplary HID button click report, that may be part of a HID command, is:
In some examples, the HID commands relating to the HID button click reports may be transmitted to the iOS computing device via a wireless interface, such as a Bluetooth interface. However, using the bandwidth limited Bluetooth connection to transmit the HID commands may result in delayed cursor clicks as viewed on the display of the iOS computing device.
The computing device then processes and reads the HID commands relating to the HID button click reports and sends instructions to the AssistiveTouch™ as a function of the HID button click reports at step 314. The AssistiveTouch™ then carries out the instructions and performs the click(s) of the pointer on the screen as a function of the HID button click reports at step 315, the click of the pointer corresponding to the click of the button of the pointer device.
The mouse click reports and the carrying out of steps 311 to 315 may be performed periodically after given time intervals.
Reference is now made to
The relay device 15 may receive peripheral information via its receiver 13 (e.g. USB bridge, or Bluetooth transceiver) from peripheral devices, such as wired/wireless keyboard 21 and/or wired/wireless mouse 22, and translate and transmit this information to computing device 30 as described herein.
The computing device 30 may also share its screen with the external screen 40 by utilizing its screen output information 35, for screen mirroring 36, the screen output information sent as data via the connector port 31 to the multiport adaptor 34, and later shared with the external screen 40 joined to computing device 30 via the multiport adaptor 34.
Moreover, in the example where the peripheral device is a wired/wireless mouse 22, the mirrored screen shown on the external screen 40 also displays the cursor, as the cursor appears and moves on the computing device's screen, by, for instance, following the steps described in
As shown in
The description of the present invention has been presented for purposes of illustration but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/CA2016/050809 | Jul 2016 | CA | national |
| PCT/CA2017/050285 | Mar 2017 | CA | national |
| PCT/CA2017/050740 | Jun 2017 | CA | national |
The present application is a continuation-in-part of international application No. PCT/CA2016/050809 filed on Jul. 11, 2016, now pending, that is a continuation-in-part of international application No PCT/CA2016/050710 filed on Jun. 17, 2016, now abandoned; is a continuation-in-part of international application No PCT/CA2017/050740 filed on Jun. 16, 2017 that is a continuation-in-part of international application No PCT/CA2017/050285 filed on Mar. 2, 2017, now pending, that claims priority of U.S. provisional No. 62/323,031 filed on Apr. 15, 2016; and claims priority of U.S. provisional application No. 62/474,012 filed on Mar. 20, 2017.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CA2017/050837 | 7/11/2017 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62474012 | Mar 2017 | US | |
| 62323031 | Apr 2016 | US | |
| 62323031 | Apr 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CA2016/050809 | Jul 2016 | US |
| Child | 16084732 | US | |
| Parent | PCT/CA2016/050710 | Jun 2016 | US |
| Child | PCT/CA2016/050809 | US | |
| Parent | PCT/CA2017/050285 | Mar 2017 | US |
| Child | PCT/CA2017/050837 | US | |
| Parent | PCT/CA2016/050809 | Jul 2016 | US |
| Child | PCT/CA2017/050285 | US | |
| Parent | PCT/CA2016/050710 | Jun 2016 | US |
| Child | PCT/CA2016/050809 | US | |
| Parent | PCT/CA2017/050740 | Jun 2017 | US |
| Child | PCT/CA2017/050837 | US | |
| Parent | PCT/CA2017/050285 | Mar 2017 | US |
| Child | PCT/CA2017/050740 | US | |
| Parent | PCT/CA2016/050710 | Jun 2016 | US |
| Child | PCT/CA2017/050285 | US | |
| Parent | PCT/CA2016/050809 | Jul 2016 | US |
| Child | PCT/CA2016/050710 | US |
