A portion of the disclosure of this patent document may contain material, which is subject to copyright protection. Certain marks referenced herein may be common law or registered trademarks of the applicant, the assignee or third parties affiliated or unaffiliated with the applicant or the assignee. Use of these marks is for providing an enabling disclosure by way of example and shall not be construed to exclusively limit the scope of the disclosed subject matter to material associated with such marks.
The invention relates to user interfaces providing an additional number of simultaneously-adjustable interactively-controlled discrete (clicks, taps, discrete gestures) and pseudo-continuous (downward pressure, roll, pitch, yaw, multi-touch geometric measurements, continuous gestures, etc.) user-adjustable settings and parameters, and in particular to the use of the USB HID device abstraction for interfacing such user interfaces to applications, and further how these can be used in applications.
The present invention provides extensions and improvements to the user interface parameter signals provided by the High Dimensional Touchpad (HTPD), for example as taught in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605, as well as other systems and methods that can incorporate similar or related technologies.
The extensions and improvements provided by the present invention include the use of the USB HID device abstraction for interfacing such user interfaces to applications.
By way of introduction, touch screens implementing tactile sensor arrays have recently received tremendous attention with the addition multi-touch sensing, metaphors, and gestures. After an initial commercial appearance in the products of FingerWorks, such advanced touch screen technologies have received great commercial success from their defining role in the iPhone and subsequent adaptations in PDAs and other types of cell phones and hand-held devices. Despite this popular notoriety and the many associated patent filings, tactile array sensors implemented as transparent touchscreens were in fact taught in the 1999 filings of issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978.
Despite the many popular touch interfaces and gestures, there remains a wide range of additional control capabilities that can yet be provided by further enhanced user interface technologies. A number of enhanced touch user interface features are described in U.S. Pat. No. 6,570,078, pending U.S. patent application Ser. Nos. 11/761,978, 12/418,605, 12/502,230, 12/541,948, and related pending US patent applications. These patents and patent applications also address popular contemporary gesture and touch features. The enhanced user interface features taught in these patents and patent applications, together with popular contemporary gesture and touch features, can be rendered by the “High Definition Touch Pad” (HDTP) technology taught in those patents and patent applications. Implementations of the HTDP provide advanced multi-touch capabilities far more sophisticated that those popularized by FingerWorks, Apple, NYU, Microsoft, Gesturetek, and others.
In an embodiment, the invention provides a user interface providing an additional number of simultaneously-adjustable interactively-controlled discrete (clicks, taps, discrete gestures) and pseudo-continuous (downward pressure, roll, pitch, yaw, multi-touch geometric measurements, continuous gestures, etc.) user-adjustable settings and parameters, this user interface further provided with a USB HID device abstraction for interfacing such user interfaces to applications.
In a first embodiment, a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor that is connected to the computer via a USB interface. Here the HDTP signal processing and HDTP gesture detection are implemented on the computer or other device.
In another embodiment, a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor and one or more associated processor(s) which in turn is/are connected to the computer via a USB interface. Here the HDTP signal processing and HDTP gesture detection are implemented on the one or more processor(s) associated with HDTP sensor
In another embodiment, a USB HID device abstraction is used as a software interface even though no USB port is actually used.
In another embodiment, a USB HID device abstraction is used to provide HDTP user interface signals to one or more applications (as well as the operating system or windowing system in some implementations).
In another embodiment, the HDTP to interface one or more applications executing on a computer or other device through use of the USB HID device class.
In another embodiment, the USB HID device class provides an open interface useful for both traditional computer pointing devices such as the standard computer mouse as well as other user interface devices such as game controllers and the Logitech 3DConnexion SpaceNavigator™.
In an embodiment, the HDTP uses one or more Report Descriptor Item(s) for creating HID protocols.
In an embodiment, the HDTP use only one set of Report Descriptor Item(s) to provide routing and mapping information for HDTP parameters and/or gestures.
In another embodiment, the HDTP uses a plurality of Report Descriptor Item(s) to provide routing and mapping information for HDTP parameters and/or gestures.
In another embodiment, the HDTP has only a single configuration and thus uses only one Configuration Descriptor.
In another embodiment, the HDTP has a plurality of configurations and thus provide a plurality of Configuration Descriptors.
In another embodiment, the HDTP includes an Interface Descriptor with class field used to define the HDTP as a HID class device.
In another embodiment, the HDTP includes boot device protocols and one or more associated HID subclasses.
In an embodiment, the HDTP includes at least host-polled communications via the “Control Pipe” formalism.
In another embodiment, the HDTP includes asynchronous communications via the “Interrupt Pipe” formalism.
In another embodiment, the HDTP includes mapping of a gesture event (symbol) stream and possible associated parameter(s) stream to corresponding USB HID messages.
In another embodiment, the USB HID messages associated with the HDTP comprise “standard” or “pseudo-standard” types of USB messages and/or other types of USB message channels.
In an embodiment, the invention comprises a method for implementing USB communications for a touch-based user interface providing user interface measurement and detection of at least one gesture and one angle of finger position, the method comprising:
In an embodiment, the method further provides for the host device to comprise a desktop computer.
In an embodiment, the method further provides for the tactile sensor array to comprise a touchscreen.
In an embodiment, the method further provides for the finger angle to comprise a yaw angle.
In an embodiment, the method further provides for the finger angle to comprise a roll angle.
In an embodiment, the method further provides for the finger angle to comprise a pitch angle.
In an embodiment, the method further provides for the gesture to comprise a finger flick.
In an embodiment, the method further provides for the processing to also produce at least one parameter associated with the gesture, the parameter comprising a value responsive to the real-time tactile-image information.
In an embodiment, the method further provides for at least one parameter associated with the gesture to be carried by the USB HID message.
In an embodiment, the method further provides for at least one of the processing, mapping, and transmitting to comprise a HID Report Descriptor.
In an embodiment, the method further provides for the HID Report Descriptor to be transmitted to the host device.
In an embodiment, the method further provides for at least one of the processing, mapping, and transmitting to comprise at least one HID Physical Descriptor.
In an embodiment, the method further provides for at least one of the processing, mapping, and transmitting to comprise at least one HID Endpoint Descriptor.
In an embodiment, the method further provides for at least one of the processing, mapping, and transmitting to comprise at least one HID Configuration Descriptor.
In an embodiment, the method further provides for the processing further recognizes a plurality of gestures.
In an embodiment, the method further provides for the processing of a selected plurality from of the gestures within the plurality of gestures also to produce at least one parameter, said parameter comprising a value responsive to real-time tactile-image information, said parameter associated with each gesture in the selected plurality.
In an embodiment, the method further provides for the value of at least one parameter associated with each gesture in the selected plurality to be carried by the USB HID message.
In an embodiment, the method further provides for a sequence of gestures to be presented to further processing to create a meta-gesture.
In an embodiment, the method further provides for the further processing to employ a tactile grammar.
20. In an embodiment, the method further provides for information representing the meta-gesture to be carried by the USB HID message.
The above and other aspects, features and advantages of the present invention will become more apparent upon consideration of the following description of preferred embodiments taken in conjunction with the accompanying drawing and figures.
a-1g depict a number of arrangements and embodiments employing the HDTP technology.
a-2e and
a is a graphical representation of a tactile image produced by contact of a human finger on a tactile sensor array.
a-10c depict camera implementations for direct viewing of at least portions of the human hand, wherein the camera image array is employed as an HDTP tactile sensor array.
a-12b depict an implementation of an arrangement comprising a video camera capturing the image of a deformable material whose image varies according to applied pressure.
a-17f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology.
a-22c depict various approaches to the handling of compound posture data images.
a depicts an embodiment wherein the raw tilt measurement is used to make corrections to the geometric center measurement under at least conditions of varying the tilt of the finger.
a-27d depict operations acting on various parameters, rates, and symbols to produce other parameters, rates, and symbols, including operations such as sample/hold, interpretation, context, etc.
a-29c depict methods for interfacing the HDTP with a browser.
a depicts a user-measurement training procedure wherein a user is prompted to touch the tactile sensor array in a number of different positions.
a (adapted from Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11) depicts the HID device class comprising a descriptor called the “HID descriptor” which in turn consists of a “Report Descriptor” and a “Physical Descriptor.”
b (adapted from combining several figures from Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11) depicts the HID class “HID Descriptor” and “Endpoint descriptor” together comprised by an “Interface Descriptor” that is in turn comprised by a “Configuration Descriptor” within the “Device Descriptor,” and (peer to the Device Descriptor) a “String Descriptor.”
c (adapted from Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11) depicts how an HID class device appears to the parser within the HID driver.
d (adapted from Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11) depicts how an HID class driver communicates with an HID class device using either host-polled communications via a “Control Pipe” formalism or an optional lower-latency asynchronous “Interrupt Pipe.”
a depicts a summary representation of the single-finger gesture recognition and associated parameter production capabilities provided for by the invention.
b depicts a summary representation of the multi-finger constellation gesture recognition and associated parameter production capabilities provided for by the invention.
The USB HID messages may comprise “standard” or “pseudo-standard” types of USB messages and/or other types of USB message channels.
a depicts the single-finger parameter channel arrangements depicted in
b depicts the single-finger parameter and gesture event arrangements depicted in
c depicts the single-finger parameter, gesture event, and associated gesture parameter arrangements depicted in
a depicts the multi-finger parameter channel arrangements depicted in
b depicts the multi-finger parameter and gesture event arrangements depicted in
c depicts the multi-finger parameter, gesture event, and associated gesture parameter arrangements depicted in
In the following detailed description, reference is made to the accompanying drawing figures which form a part hereof, and which show by way of illustration specific embodiments of the invention. It is to be understood by those of ordinary skill in this technological field that other embodiments can be utilized, and structural, electrical, as well as procedural changes can be made without departing from the scope of the present invention. Wherever possible, the same element reference numbers will be used throughout the drawings to refer to the same or similar parts.
Despite the many popular touch interfaces and gestures in contemporary information appliances and computers, there remains a wide range of additional control capabilities that can yet be provided by further enhanced user interface technologies. A number of enhanced touch user interface features are described in U.S. Pat. No. 6,570,078, pending U.S. patent application Ser. Nos. 11/761,978, 12/418,605, 12/502,230, 12/541,948, and related pending US patent applications. These patents and patent applications also address popular contemporary gesture and touch features. The enhanced user interface features taught in these patents and patent applications, together with popular contemporary gesture and touch features, can be rendered by the “High Definition Touch Pad” (HDTP) technology taught in those patents and patent applications.
The present patent application addresses additional technologies for feature and performance improvements of HDTP technologies. Specifically, this patent application providing and/or implementing HDTP technologies with a USB HID device abstraction for interfacing such user interfaces to applications.
Before providing details specific to the present invention, some embodiments of HDTP technology are provided. This will be followed by a summarizing overview of HDTP technology.
Exemplary Embodiments Employing a Touchpad and Touchscreen Form of a HDTP
a-1g and 2a-2e depict a number of arrangements and embodiments employing the HDTP technology.
e depicts an HDTP integrated into a cell phone, smartphone, PDA, or other hand-held consumer device.
g depicts an HDTP touchscreen configuration that can be used in a tablet computer, wall-mount computer monitor, digital television, video conferencing screen, kiosk, etc.
In at least the arrangements of
Embodiments Incorporating the HDTP into a Traditional or Contemporary Generation Mouse
a-2e and
In the integrations depicted in
In another embodiment taught in the specification of issued U.S. Pat. No. 7,557,797 and associated pending continuation applications more than two touchpads can be included in the advance mouse embodiment, for example as suggested in the arrangement of
The information in this section provides an overview of HDTP user interface technology as described in U.S. Pat. No. 6,570,078, pending U.S. patent application Ser. Nos. 11/761,978, 12/418,605, 12/502,230, 12/541,948, and related pending US patent applications.
In an embodiment, a touchpad used as a pointing and data entry device can comprise an array of sensors. The array of sensors is used to create a tactile image of a type associated with the type of sensor and method of contact by the human hand.
In one embodiment, the individual sensors in the sensor array are pressure sensors and a direct pressure-sensing tactile image is generated by the sensor array.
In another embodiment, the individual sensors in the sensor array are proximity sensors and a direct proximity tactile image is generated by the sensor array. Since the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the sensor array comprised of proximity sensors also provides an indirect pressure-sensing tactile image.
In another embodiment, the individual sensors in the sensor array can be optical sensors. In one variation of this, an optical image is generated and an indirect proximity tactile image is generated by the sensor array. In another variation, the optical image can be observed through a transparent or translucent rigid material and, as the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the optical sensor array also provides an indirect pressure-sensing tactile image.
In some embodiments, the array of sensors can be transparent or translucent and can be provided with an underlying visual display element such as an alphanumeric and/or graphics and/or image display. The underlying visual display can comprise, for example, an LED array display, a backlit LCD, etc. Such an underlying display can be used to render geometric boundaries or labels for soft-key functionality implemented with the tactile sensor array, to display status information, etc. Tactile array sensors implemented as transparent touchscreens are taught in the 1999 filings of issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978.
In an embodiment, the touchpad or touchscreen can comprise a tactile sensor array obtains or provides individual measurements in every enabled cell in the sensor array that provides these as numerical values. The numerical values can be communicated in a numerical data array, as a sequential data stream, or in other ways. When regarded as a numerical data array with row and column ordering that can be associated with the geometric layout of the individual cells of the sensor array, the numerical data array can be regarded as representing a tactile image. The only tactile sensor array requirement to obtain the full functionality of the HDTP is that the tactile sensor array produce a multi-level gradient measurement image as a finger, part of hand, or other pliable object varies is proximity in the immediate area of the sensor surface.
Such a tactile sensor array should not be confused with the “null/contact” touchpad which, in normal operation, acts as a pair of orthogonally responsive potentiometers. These “null/contact” touchpads do not produce pressure images, proximity images, or other image data but rather, in normal operation, two voltages linearly corresponding to the location of a left-right edge and forward-back edge of a single area of contact. Such “null/contact” touchpads, which are universally found in existing laptop computers, are discussed and differentiated from tactile sensor arrays in issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978 (pre-grant publication US 2007/0229477). Before leaving this topic, it is pointed out that these the “null/contact” touchpads nonetheless can be inexpensively adapted with simple analog electronics to provide at least primitive multi-touch capabilities as taught in issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978 (therein, paragraphs [0022]-[0029], for example).
More specifically,
In many various embodiments, the tactile sensor array can be connected to interface hardware that sends numerical data responsive to tactile information captured by the tactile sensor array to a processor. In various embodiments, this processor will process the data captured by the tactile sensor array and transform it various ways, for example into a collection of simplified data, or into a sequence of tactile image “frames” (this sequence akin to a video stream), or into highly refined information responsive to the position and movement of one or more fingers and/or other parts of the hand.
As to further detail of the latter example, a “frame” refers to a 2-dimensional list, number of rows by number of columns, of tactile measurement value of every pixel in a tactile sensor array at a given instance. The time interval between one frame and the next one depends on the frame rate of the system and the number of frames in a unit time (usually frames per second).
Exemplary Types of Tactile Sensor Arrays
The tactile sensor array employed by HDTP technology can be implemented by a wide variety of means, for example:
An example implementation of a tactile sensor array is a pressure sensor array. Pressure sensor arrays discussed in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978.
Capacitive proximity sensors can be used in various handheld devices with touch interfaces (see for example, among many, http://electronics.howstuffworks.com/iphone2.htm, http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf). Prominent manufacturers and suppliers of such sensors, both in the form of opaque touchpads and transparent touch screens, include Balda AG (Bergkirchener Str. 228, 32549 Bad Oeynhausen, Del., www.balda.de), Cypress (198 Champion Ct., San Jose, Calif. 95134, www.cypress.com), and Synaptics (2381 Bering Dr., San Jose, Calif. 95131, www.synaptics.com). In such sensors, the region of finger contact is detected by variations in localized capacitance resulting from capacitive proximity effects induced by an overlapping or otherwise nearly-adjacent finger. More specifically, the electrical field at the intersection of orthogonally-aligned conductive buses is influenced by the vertical distance or gap between the surface of the sensor array and the skin surface of the finger. Such capacitive proximity sensor technology is low-cost, reliable, long-life, stable, and can readily be made transparent.
Forrest M. Mims is credited as showing that an LED can be used as a light detector as well as a light emitter. Recently, light-emitting diodes have been used as a tactile proximity sensor array (for example, as depicted in the video available at http://cs.nyu.edu/˜jhan/ledtouch/index.html). Such tactile proximity array implementations typically need to be operated in a darkened environment (as seen in the video in the above web link). In one embodiment provided for by the invention, each LED in an array of LEDs can be used as a photodetector as well as a light emitter, although a single LED can either transmit or receive information at one time. Each LED in the array can sequentially be selected to be set to be in receiving mode while others adjacent to it are placed in light emitting mode. A particular LED in receiving mode can pick up reflected light from the finger, provided by said neighboring illuminating-mode LEDs.
Use of video cameras for gathering control information from the human hand in various ways is discussed in U.S. Pat. No. 6,570,078 and Pending U.S. patent application Ser. No. 10/683,915. Here the camera image array is employed as an HDTP tactile sensor array. Images of the human hand as captured by video cameras can be used as an enhanced multiple-parameter interface responsive to hand positions and gestures, for example as taught in U.S. patent application Ser. No. 10/683,915 Pre-Grant-Publication 2004/0118268 (paragraphs [314], [321]-[332], [411], [653], both stand-alone and in view of [325], as well as [241]-[263]).
In another video camera tactile controller embodiment, a flat or curved transparent or translucent surface or panel can be used as sensor surface. When a finger is placed on the transparent or translucent surface or panel, light applied to the opposite side of the surface or panel reflects light in a distinctly different manner than in other regions where there is no finger or other tactile contact. The image captured by an associated video camera will provide gradient information responsive to the contact and proximity of the finger with respect to the surface of the translucent panel. For example, the parts of the finger that are in contact with the surface will provide the greatest degree of reflection while parts of the finger that curve away from the surface of the sensor provide less reflection of the light. Gradients of the reflected light captured by the video camera can be arranged to produce a gradient image that appears similar to the multilevel quantized image captured by a pressure sensor. By comparing changes in gradient, changes in the position of the finger and pressure applied by the finger can be detected.
a-12b depict an implementation of an arrangement comprising a video camera capturing the image of a deformable material whose image varies according to applied pressure. In the example of
Compensation for Non-Ideal Behavior of Tactile Sensor Arrays
Individual sensor elements in a tactile sensor array produce measurements that vary sensor-by-sensor when presented with the same stimulus. Inherent statistical averaging of the algorithmic mathematics can damp out much of this, but for small image sizes (for example, as rendered by a small finger and/or light contact), as well as in cases where there are extremely large variances in sensor element behavior from sensor to sensor, the invention provides for each sensor to be individually calibrated in implementations where that can be advantageous. Sensor-by-sensor measurement value scaling, offset, and/or nonlinear warpings can be invoked for all or selected sensor elements during data acquisition scans. Similarly, the invention provides for individual noisy or defective sensors can be tagged for omission during data acquisition scans.
Additionally, the macroscopic arrangement of sensor elements can introduce nonlinear spatial warping effects. As an example, various manufacturer implementations of capacitive proximity sensor arrays and associated interface electronics are known to comprise often dramatic nonlinear spatial warping effects.
Exemplary Types of Hand Contact Measurements and Features Provided by HDTP Technology
a-17f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology. The depiction in these figures is from the side of the touchpad.
Each of the six parameters listed above can be obtained from operations on a collection of sums involving the geometric location and tactile measurement value of each tactile measurement sensor. Of the six parameters, the left-right geometric center (“x”), forward-back geometric center (“y”), and clockwise-counterclockwise yaw rotation (“ψ”) can be obtained from binary threshold image data. The average downward pressure (“p”), roll (“φ”), and pitch (“θ”) parameters are in some embodiments beneficially calculated from gradient (multi-level) image data. One remark is that because binary threshold image data is sufficient for the left-right geometric center, forward-back geometric center, and clockwise-counterclockwise yaw rotation parameters, these also can be discerned for flat regions of rigid non-pliable objects, and thus the HDTP technology thus can be adapted to discern these three parameters from flat regions with striations or indentations of rigid non-pliable objects.
These ‘Position Displacement’ parameters
Each of the six parameters portrayed in
The HDTP technology provides for multiple points of contact, these days referred to as “multi-touch.”
By way of example,
HDTP technology robustly provides feature-rich capability for tactile sensor array contact with two or more fingers, with other parts of the hand, or with other pliable (and for some parameters, non-pliable) objects. In one embodiment, one finger on each of two different hands can be used together to at least double number of parameters that can be provided. Additionally, new parameters particular to specific hand contact configurations and postures can also be obtained. By way of example,
Other HDTP Processing, Signal Flows, and Operations
In order to accomplish this range of capabilities, HDTP technologies must be able to parse tactile images and perform operations based on the parsing. In general, contact between the tactile-sensor array and multiple parts of the same hand forfeits some degrees of freedom but introduces others. For example, if the end joints of two fingers are pressed against the sensor array as in
In general, compound images can be adapted to provide control over many more parameters than a single contiguous image can. For example, the two-finger postures considered above can readily pro-vide a nine-parameter set relating to the pair of fingers as a separate composite object adjustable within an ergonomically comfortable range. One example nine-parameter set the two-finger postures consider above is:
As another example, by using the whole hand pressed flat against the sensor array including the palm and wrist, it is readily possible to vary as many as sixteen or more parameters independently of one another. A single hand held in any of a variety of arched or partially-arched postures provides a very wide range of postures that can be recognized and parameters that can be calculated.
When interpreted as a compound image, extracted parameters such as geometric center, average downward pressure, tilt (pitch and roll), and pivot (yaw) can be calculated for the entirety of the asterism or constellation of smaller blobs. Additionally, other parameters associated with the asterism or constellation can be calculated as well, such as the aforementioned angle of separation between the fingers. Other examples include the difference in downward pressure applied by the two fingers, the difference between the left-right (“x”) centers of the two fingertips, and the difference between the two forward-back (“y”) centers of the two fingertips. Other compound image parameters are possible and are provided by HDTP technology.
There are number of ways for implementing the handling of compound posture data images. Two contrasting examples are depicted in
b depicts an alternative embodiment, tactile image data is examined for the number M of isolated blobs (“regions”) and the primitive running sums are calculated for each blob, but this information is directed to a multi-regional tactile image parameter extraction stage. Such a stage can include, for example, compensation for minor or major ergonomic interactions among the various degrees of postures of the hand. The resulting compensation or otherwise produced extracted parameter sets (for example, x position, y position, average pressure, roll, pitch, yaw) uniquely associated with each of the M blobs and total number of blobs are directed to a compound image parameter mapping function to produce various types of outputs as described for the arrangement of
Additionally, embodiments of the invention can be set up to recognize one or more of the following possibilities:
Embodiments that recognize two or more of these possibilities can further be able to discern and process combinations of two more of the possibilities.
c depicts a simple system for handling one, two, or more of the above listed possibilities, individually or in combination. In the general arrangement depicted, tactile sensor image data is analyzed (for example, in the ways described earlier) to identify and isolate image data associated with distinct blobs. The results of this multiple-blob accounting is directed to one or more global classification functions set up to effectively parse the tactile sensor image data into individual separate blob images and/or individual compound images. Data pertaining to these individual separate blob and/or compound images are passed on to one or more parallel and/or serial parameter extraction functions. The one or more parallel and/or serial parameter extraction functions can also be provided information directly from the global classification function(s). Additionally, data pertaining to these individual separate blob and/or compound images are passed on to additional image recognition function(s), the output of which can also be provided to one or more parallel and/or serial parameter extraction function(s). The output(s) of the parameter extraction function(s) can then be either used directly, or first processed further by parameter mapping functions. Clearly other implementations are also possible to one skilled in the art and these are provided for by the invention.
Refining of the HDTP User Experience
As an example of user-experience correction of calculated parameters, it is noted that placement of hand and wrist at a sufficiently large yaw angle can affect the range of motion of tilting. As the rotation angle increases in magnitude, the range of tilting motion decreases as mobile range of human wrists gets restricted. The invention provides for compensation for the expected tilt range variation as a function of measured yaw rotation angle. An embodiment is depicted in the middle portion of
As the finger is tilted to the left or right, the shape of the area of contact becomes narrower and shifts away from the center to the left or right. Similarly as the finger is tilted forward or backward, the shape of the area of contact becomes shorter and shifts away from the center forward or backward. For a better user experience, the invention provides for embodiments to include systems and methods to compensate for these effects (i.e. for shifts in blob size, shape, and center) as part of the tilt measurement portions of the implementation. Additionally, the raw tilt measures can also typically be improved by additional processing.
Additional HDTP Processing, Signal Flows, and Operations
The HDTP affords and provides for yet further capabilities. For example, sequence of symbols can be directed to a state machine, as shown in
In an arrangement such as the one of
Alternatively, these two cursor-control parameters can be provided by another user interface device, for example another touchpad or a separate or attached mouse.
In some situations, control of the cursor location can be implemented by more complex means. One example of this would be the control of location of a 3D cursor wherein a third parameter must be employed to specify the depth coordinate of the cursor location. For these situations, the arrangement of
Focus control is used to interactively routing user interface signals among applications. In most current systems, there is at least some modality wherein the focus is determined by either the current cursor location or a previous cursor location when a selection event was made. In the user experience, this selection event typically involves the user interface providing an event symbol of some type (for example a mouse click, mouse double-click touchpad tap, touchpad double-tap, etc). The arrangement of
In some embodiments, each application that is a candidate for focus selection provides a window displayed at least in part on the screen, or provides a window that can be deiconified from an icon tray or retrieved from beneath other windows that can be obfuscating it. In some embodiments, if the background window is selected, focus selection element that directs all or some of the broader information stream from the HDTP system to the operating system, window system, and/or features of the background window. In some embodiments, the background window can be in fact regarded as merely one of the applications shown in the right portion of the arrangement of
Use of the Additional HDTP Parameters by Applications
The types of human-machine geometric interaction between the hand and the HDTP facilitate many useful applications within a visualization environment. A few of these include control of visualization observation viewpoint location, orientation of the visualization, and controlling fixed or selectable ensembles of one or more of viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, simulation control parameters, etc. As one example, the 6D orientation of a finger can be naturally associated with visualization observation viewpoint location and orientation, location and orientation of the visualization graphics, etc. As another example, the 6D orientation of a finger can be naturally associated with a vector field orientation for introducing synthetic measurements in a numerical simulation.
As yet another example, at least some aspects of the 6D orientation of a finger can be naturally associated with the orientation of a robotically positioned sensor providing actual measurement data. As another example, the 6D orientation of a finger can be naturally associated with an object location and orientation in a numerical simulation. As another example, the large number of interactive parameters can be abstractly associated with viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, numeric simulation control parameters, etc.
In yet another example, the x and y parameters provided by the HDTP can be used for focus selection and the remaining parameters can be used to control parameters within a selected GUI.
In still another example, the x and y parameters provided by the HDTP can be regarded as a specifying a position within an underlying base plane and the roll and pitch angles can be regarded as a specifying a position within a superimposed parallel plane. In a first extension of the previous two-plane example, the yaw angle can be regarded as the rotational angle between the base and superimposed planes. In a second extension of the previous two-plane example, the finger pressure can be employed to determine the distance between the base and superimposed planes. In a variation of the previous two-plane example, the base and superimposed plane can not be fixed as parallel but rather intersect as an angle associated with the yaw angle of the finger. In the each of these, either or both of the two planes can represent an index or indexed data, a position, pair of parameters, etc. of a viewing aspect, visualization rendering aspect, pre-visualization operations, data selection, numeric simulation control, etc.
A large number of additional approaches are possible as is appreciated by one skilled in the art. These are provided for by the invention.
Support for Additional Parameters Via Browser Plug-Ins
The additional interactively-controlled parameters provided by the HDTP provide more than the usual number supported by conventional browser systems and browser networking environments. This can be addressed in a number of ways.
In a first approach, an HDTP interfaces with a browser both in a traditional way and additionally via a browser plug-in. Such an arrangement can be used to capture the additional user interface input parameters and pass these on to an application interfacing to the browser. An example of such an arrangement is depicted in
In a second approach, an HDTP interfaces with a browser in a traditional way and directs additional GUI parameters though other network channels. Such an arrangement can be used to capture the additional user interface input parameters and pass these on to an application interfacing to the browser. An example of such an arrangement is depicted in
In a third approach, an HDTP interfaces all parameters to the browser directly. Such an arrangement can be used to capture the additional user interface input parameters and pass these on to an application interfacing to the browser. An example of such an arrangement is depicted in
The browser can interface with local or web-based applications that drive the visualization and/or control the data source(s), process the data, etc. The browser can be provided with client-side software such as JAVA Script. The browser can provide also be configured advanced graphics to be rendered within the browser display environment, allowing the browser to be used as a viewer for data visualizations, advanced animations, etc., leveraging the additional multiple parameter capabilities of the HDTP. The browser can interface with local or web-based applications that drive the advanced graphics. In an embodiment, the browser can be provided with Simple Vector Graphics (“SVG”) utilities (natively or via an SVG plug-in) so as to render basic 2D vector and raster graphics. In another embodiment, the browser can be provided with a 3D graphics capability, for example via the Cortona 3D browser plug-in.
Multiple Parameter Extensions to Traditional Hypermedia Objects
The HDTP can be used to provide extensions to the traditional and contemporary hyperlink, roll-over, button, menu, and slider functions found in web browsers and hypermedia documents leveraging additional user interface parameter signals provided by an APD (i.e., HTPD, Advanced Mice, and other rich parameter user interfaces including currently popular advanced touch interfaces employing multitouch and/or gestures). The extensions provided by the invention include:
Potential uses of the MHOs and more generally extensions provided for by the invention include:
In addition to MHOs that are additional-parameter extensions of traditional hypermedia objects, new types of MHOs unlike traditional or contemporary hypermedia objects can be implemented leveraging the additional user interface parameter signals and user interface metaphors that can be associated with them. Illustrative examples include:
In any of these, the invention provides for the MHO to be activated or selected by various means, for example by clicking or tapping when the cursor is displayed within the area, simply having the cursor displayed in the area (i.e., without clicking or tapping, as in rollover), etc.
It is anticipated that variations on any of these and as well as other new types of MHOs can similarly be crafted by those skilled in the art and these are provided for by the invention.
User Training
Since there is a great deal of variation from person to person, it is useful to include a way to train the invention to the particulars of an individual's hand and hand motions. For example, in a computer-based application, a measurement training procedure will prompt a user to move their finger around within a number of different positions while it records the shapes, patterns, or data derived from it for later use specifically for that user.
Typically most finger postures make a distinctive pattern. In one embodiment, a user-measurement training procedure could involve having the user prompted to touch the tactile sensor array in a number of different positions, for example as depicted in
The range in motion of the finger that can be measured by the sensor can subsequently be re-corded in at least two ways. It can either be done with a timer, where the computer will prompt user to move his finger from position 3000 to position 3001, and the tactile image imprinted by the finger will be recorded at points 3001.3, 3001.2 and 3001.1. Another way would be for the computer to query user to tilt their finger a portion of the way, for example “Tilt your finger ⅔ of the full range” and record that imprint. Other methods are clear to one skilled in the art and are provided for by the invention.
Additionally, this training procedure allows other types of shapes and hand postures to be trained into the system as well. This capability expands the range of contact possibilities and applications considerably. For example, people with physical handicaps can more readily adapt the system to their particular abilities and needs.
Multitouch Architecture
Additional Parameter Refinement
Additional refinement of the parameters can then be obtained by additional processing. As an example,
USB HID and Other Interfacing to Host Computer or Other Devices
In certain embodiments use of a USB interface in an HDTP implementation is useful, desirable, or required.
It is noted that although this section is directed towards various example implementations involving the Universal Serial Bus (USB), this section address more generally presenting HDTP technology to the rest of the computer system in standardized manner. For example:
In a first example embodiment, an HDTP sensor that is connected to a computer or other device via an USB interface. Here the HDTP signal processing and any HDTP gesture processing are implemented on the hosting computer or other device. The HDTP signal processing and any HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
In second example embodiment, a USB HID device abstraction is employed to connect a host computer or other device with an HDTP sensor and one or more associated processor(s) which in turn is/are connected to the host computer via a USB interface. Here the HDTP signal processing and any HDTP gesture detection are implemented on the one or more processor(s) associated with HDTP sensor. The HDTP signal processing and any HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
In a third example embodiment, a USB HID device abstraction is used as a software interface even though no USB port is actually used. Such an implementation is useful in cases where the HDTP is fully integrated into the host computer or other device, for example as in the case of a laptop computer, tablet computer, smartphone, etc. The HDTP signal processing and any HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
In the case of the first example embodiment, the USB interface could, for example, be used to transport a tactile image or other pre-processed information. In the case of the second and third example embodiment, the invention provides for a USB HID device abstraction is used to provide HDTP user interface signals to one or more applications (as well as the operating system or windowing system in some implementations).
The USB HID device class provides an open interface useful for both traditional computer pointing devices such as the standard computer mouse as well as other user interface devices such as game controllers and the Logitech 3DConnexion SpaceNavigator™. The invention provides for the HDTP to interface one or more applications executing on a computer or other device through use of the USB HID device class.
As taught in the Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11, Section 3 (p. 4), information associated with a USB device comprises information “segments” called “Descriptors” which are used to identify a device as belonging to one of a collection of “classes.” The USB HID device class is used to identify and specify devices serving or performing as “Human Interface Devices” (HID). The USB HID device class is currently specified at the time of this patent application by at least the Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID) Version 1.11 (Jun. 6, 2001). Some example HID implementations for various example peripheral devices are provided in Universal Serial Bus (USB) HID Usage Tables, Version 1.12 (Oct. 28, 2004), currently available at the time of the patent application filing at the URL http://www.usb.org/developers/devclass_docs/Hut1—12v2.pdf.
The HID device class comprises a descriptor called the “HID Descriptor” which in turn consists of a “Physical Descriptor Set” and a “Report Descriptor,” the Report Descriptor in turn comprising one or more “Item(s)” as shown in
In various embodiments, the HDTP communicating via USB HID could be configured to act as various types of devices communicate various events and parameters to a host computer or other device. The exact definition of each candidate device is implemented via Report Descriptors. There are established Report Descriptors such as for those for common devices like mouse, keyboard and game controllers, and custom devices can also readily be defined. Example fields within the Report Descriptors that are already supported are listed in the aforementioned Universal Serial Bus (USB) HID Usage Tables, Version 1.12. Some examples relevant to the HDTP include:
The arrangement associated with
The Interface Descriptor also has broader roles in the support of various USB devices and implementations, but in the case of HID devices the class field of the Interface Descriptor is used to define the peripheral device as a HID class device. The invention provides for selected HDTP embodiments to include an Interface Descriptor with class field used to define the HDTP as a HID class device.
The HID specification also include notions of subclasses and subclass protocols, but typically these are problematic and by default the Report Descriptor is typically used for creating protocols for existing and new human interface devices. The invention provides for selected HDTP embodiments to use one or more Report Descriptor Item(s) for creating HID protocols.
The subclass formalism is typically used for devices involved in machine booting operations (such as BIOS), the subclass relating to predefined protocols such as those for standard keyboards and mice. The invention provides for selected HDTP embodiments to include boot device protocols and one or more associated HID subclasses.
The HID class driver, depicted earlier in
As shown in
As described earlier in conjunction with
As taught earlier in conjunction with
Additionally, as taught in U.S. Pat. No. 6,570,078, embodiments of the HDTP can recognize and provide rich metaphor capabilities and other arrangements which involve combinations of two or more independent simultaneous gestures. In some cases the two or more independent simultaneous gestures may be rendered with separate fingers (for example as taught in conjunction with
The capabilities described in conjunction with
Further, the USB HID messages associated with some embodiments of the HDTP can comprise “standard” or “pseudo-standard” types of USB messages and/or other types of USB message channels. For example, in some embodiments, the HDTP can use standard messages used for mouse and/or keyboard (as described a few paragraphs above). As another example, the HDTP can use (more loosely) standardized messages used for the existing multi-axis game controller HID report descriptors and profiles. As another example, the HDTP can use the arrangement and messages employed by the Logitech 3DConnexion SpaceNavigator™ as a pseudo-standard. This allows the HDTP to operate the large number of commercial 3D software applications already supporting the Logitech 3DConnexion SpaceNavigator™ (see for example the list at http://www.3dconnexion.com/supported-software/software0.html, visited Jan. 22, 2011) so as to provide the HDTP's highly-improved user experience, ease-of-use, rich metaphors, and superior precision-control performance to those commercial 3D software applications.
As an example of merely one of the many possibilities,
a depicts the single-finger parameter channel arrangements depicted in
b depicts the single-finger parameter and gesture event arrangements depicted in
c depicts the single-finger parameter, gesture event, and associated gesture parameter arrangements depicted in
a depicts the multi-finger parameter channel arrangements depicted in
b depicts the multi-finger parameter and gesture event arrangements depicted in
c depicts the multi-finger parameter, gesture event, and associated gesture parameter arrangements depicted in
The terms “certain embodiments”, “an embodiment”, “embodiment”, “embodiments”, “the embodiment”, “the embodiments”, “one or more embodiments”, “some embodiments”, and “one embodiment” mean one or more (but not all) embodiments unless expressly specified otherwise. The terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless expressly specified otherwise. The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms “a”, “an” and “the” mean “one or more”, unless expressly specified otherwise.
While the invention has been described in detail with reference to disclosed embodiments, various modifications within the scope of the invention will be apparent to those of ordinary skill in this technological field. It is to be appreciated that features described with respect to one embodiment typically can be applied to other embodiments.
The invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Although exemplary embodiments have been provided in detail, various changes, substitutions and alternations could be made thereto without departing from spirit and scope of the disclosed subject matter as defined by the appended claims. Variations described for the embodiments may be realized in any combination desirable for each particular application. Thus particular limitations and embodiment enhancements described herein, which may have particular advantages to a particular application, need not be used for all applications. Also, not all limitations need be implemented in methods, systems, and apparatuses including one or more concepts described with relation to the provided embodiments. Therefore, the invention properly is to be construed with reference to the claims.
Pursuant to 35 U.S.C. §119(e), this application claims benefit of priority from Provisional U.S. Patent application Ser. No. 61/435,401, filed Jan. 24, 2011, the contents of which are incorporated by reference.
Number | Date | Country | |
---|---|---|---|
61435401 | Jan 2011 | US |