Patent Application
20080154573
Often with new input devices, there is little or no actual platform support for such an input device when it is first introduced. Hence, developing and testing software for such a platform becomes extremely difficult. For example, developers may desire to test a 64-bit tablet operating system (OS) for use with a digital pen on a 64-bit tablet personal computer (PC). However, 64-bit tablet PCs are currently not available. This situation hinders the software developer community from developing and releasing applications for 64-bit tablet PCs in a timely manner.
The following presents a simplified summary of the disclosure in order to provide a basic understanding to the reader. This summary is not an extensive overview of the disclosure and it does not identify key/critical elements of the invention or delineate the scope of the invention. Its sole purpose is to present some concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.
Embodiments herein simulate a new input device using one or more old input devices. Input data from an old input device are morphed into the input data of a new input device. In one example, a mouse and a keyboard may be used to simulate a digital pen and tablet PC buttons. The new input device data may be provided to a simulation system for injection into an operating system device stack.
Many of the attendant features will be more readily appreciated as the same becomes better understood by reference to the following detailed description considered in connection with the accompanying drawings.
Like reference numerals are used to designate like parts in the accompanying drawings.
The detailed description provided below in connection with the appended drawings is intended as a description of the present examples and is not intended to represent the only forms in which the present examples may be constructed or utilized. The description sets forth the functions of the examples and the sequence of steps for constructing and operating the examples. However, the same or equivalent functions and sequences may be accomplished by different examples.
Although not required, embodiments of the invention will be described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media (discussed below). Computer readable instructions may be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.
Additionally, device 100 may also have additional features and/or functionality. For example, device 100 may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in
The term “computer readable media” as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Memory 104 and storage 108 are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by device 100. Any such computer storage media may be part of device 100.
The term “computer readable media” may include communication media. Device 100 may also include communication connection(s) 112 that allow the device 100 to communicate with other devices, such as with other computing devices through network 120. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media.
Device 100 may also have input device(s) 114 such as keyboard, mouse, pen, voice input device, touch input device, laser range finder, infra-red cameras, video input devices, and/or any other input device. Output device(s) 116 such as one or more displays, speakers, printers, and/or any other output device may also be included. Input devices 114 and output devices 116 may be coupled to the computing device 100 via a wired connection, wireless connection, or any combination thereof. In the following description and claims, the term “coupled” and its derivatives may be used. “Coupled” may mean that two or more elements are in contact (physically, electrically, magnetically, optically, etc.). “Coupled” may also mean two or more elements are not in contact with each other, but still cooperate or interact with each other.
Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, a computing device 130 accessible via network 120 may store computer readable instructions to implement one or more embodiments of the invention. Computing device 100 may access computing device 130 and download a part or all of the computer readable instructions for execution. Alternatively, computing device 100 may download pieces of the computer readable instructions, as needed, or some instructions may be executed at computing device 100 and some at computing device 130. Those skilled in the art will also realize that all or a portion of the computer readable instructions may be carried out by a dedicated circuit, such as a Digital Signal Processor (DSP), programmable logic array, and the like.
Turning to
Morphing architecture 202 captures old input device data and morphs the input data into new input device data. Morphing architecture 202 may then hand the new input device data to new input device simulation system 214 for injection into the OS device stack 220. It will be appreciated that morphing architecture 202 is a pluggable component. Morphing architecture 202 may be used with any new input device simulation system.
In one embodiment, morphing architecture 202 is an application executing on computing device 200. Morphing architecture 202 captures the old input device data as it is received from old input device 204. However, no other applications act on the old device input data because morphing architecture eats up the old device input data (discussed further below). The old input device data is then morphed into new input device data and injected into OS device stack 220 using new input device simulation system 214. Other applications and/or the OS may then act on the new input device data.
In the case where the old input device data is not to be morphed, then the old input device data is not “eaten” but allowed to enter OS device stack 220 and eventually reach any interested consumers, such as a user application.
Morphing architecture 202 receives input data from an old input device 204. Logic of the morphing architecture (shown at 208) determines if the input data is to be treated as old input device data and injected as is into OS device stack 220 or if the input data is to be morphed into new input device data before being injected into OS device stack 220. New input device 206 is shown with a dotted line since new input device 206 is not actually attached to the computing device or new input device 206 may not even exist yet. Old input device 204 includes any input device actually coupled to computing device 200.
Morphing architecture 202 includes a capture component 210 and a generation component 212. Capture component 210 is used to capture, in real-time, data coming from usage of old input device 204. Capture component 210 may map the data captured from old input device 204 to the corresponding data elements of new input device 206.
Generation component 212 receives the old input device data from capture component 210 and generates input data for new input device 206. At least a portion of the input data for new input device 206 is based on the input data from old input device 204. In some embodiments, generation component 212 may generate input data for new input device 206 that have no corresponding elements in old input device 204.
Morphing architecture 202 provides the new input device data to new input device simulation system 214 for injection into OS device stack 220. OS device stack 220 may then act on the data as if it came from new input device 206.
Turning to
In one embodiment, an input from old input device 204 is used to signal the morphing architecture to toggle between old input device 204 and new input device 206. For example, when a keyboard/mouse is used as old input devices, a particular key, such as F1, may be used to toggle between keyboard/mouse and simulation of a new input device. In another example, an icon in a user interface allows the user to toggle between the mouse as simulating a new input device or as a conventional mouse.
In block 306, the old input device data is captured. In one embodiment, capture component 210 eats the input data from old input device 204 when capturing the input data so that other components listening for old input device data do not “hear” the old input device data. In one example, a chain of consumers may be listening for input data from the old input device. Traditionally, a consumer reads the data before passing the input data on to another consumer in the chain. Capture component 210 may inject itself at the beginning of the chain. When capture component 210 eats the old device input data, other consumers down the chain do not see the input data from the old input device and thus not realize activity occurred at the old input device.
Continuing to block 308, the old input device data is mapped to an action by the new input device. For example, when simulating a digital pen with a mouse and a keyboard, a left click received from the mouse maps to a pen tip down action by the digital pen.
Continuing to block 310, new input device data is generated based at least in part on the old input device data. In one embodiment, generation component 212 may include a set of APIs for generating the new input device data. In another embodiment, a new input device data element may be generated that does not have a corresponding old input device data element. For example, embodiments of the invention may be used to simulate a digital pen using a mouse and keyboard. However, input from the digital pen may include data elements such as pen pressure. Since the mouse or keyboard do not have a corresponding pressure element, a pen pressure data element is created in order to complete the input data from the digital pen.
Continuing block 312, the new input device data is used to simulate an input from the new input device. In the embodiment of
Continuing to block 314, the new input device data is injected into the OS device stack by the new input device simulation system. In one embodiment, the new device input data is injected into the bottom layer of OS device stack 220. The input data enters the stack at the same level and with the same properties as if the input data was coming from a real hardware device. From that point onwards, the input data is treated as the new input device data and travels up the stack. In an example of simulating a digital pen, the digital pen data travels up a tablet personal computer (PC) software stack (also referred to as an inking stack) to be converted into ink, strokes, gestures, words, etc.
Turning to
A tablet PC may use a digital pen as an input device and ink as a native data type in an operating system platform. A digital pen has specific input properties. Such properties of a digital pen include pen location, pen pressure, pen tilt, and pen button state. A tablet PC may use these pen properties to provide pen gestures, pen feedback, digital ink, handwriting recognition, and the like.
Turning to
Embodiments of tablet PC 502 may include a convertible model tablet PC that has an attached keyboard and may appear as a conventional notebook computer. However, the screen may be rotated and folded down to lie flat over the keyboard. Embodiments of tablet PC 502 may also include a personal digital assistant, a mobile phone, or other computing devices that include a screen that may be interacted with using a digital pen.
Digital pen 504 may include a pen tip 510, a pen barrel button 512, and a digital eraser 514. Pen barrel button 512 may have pre-defined functionality and/or user-defined functionality. A single pen barrel button 512 is shown for clarity but alternative embodiments may include additional pen buttons.
Tablet PC 502 may include tablet PC buttons, such as buttons 521-524. In one embodiment, tablet PC buttons 521-524 are hardware buttons that may have pre-defined functionality and/or user-defined functionality. While tablet PC 502 shows four tablet PC buttons, alternative embodiments may include more or less than four buttons. Embodiments of morphing keyboard inputs into tablet PC button inputs are discussed below in conjunction with
Referring again to
Morphing architecture 402 receives input data from keyboard 404 and/or mouse 405. Logic 408 of morphing architecture 402 determines if the input data is to behave as a digital pen. If the answer is no, then the input data is injected into OS device stack 220 has old input device data. If the answer is yes, then the old input device data is captured by stylomouse 410.
Stylomouse 410 maps the keyboard and/or mouse input data to a pen action. A pen action may include a stroke or a gesture. When the pen actions are injected into OS device stack 220, a recognizer component interprets the pen action as a stroke or a gesture.
In one embodiment, stylomouse 410 may map the mouse and/or keyboard inputs to properties of a stroke. A stroke may be made on the tablet PC using the digital pen. In one embodiment, a stroke is defined as the set of data associated in a single pen-down, pen-move, and pen-up sequence. The stroke data includes a collection of packets. A packet is the set of data the digitizer beneath the tablet PC screen detects at each sample point. Stroke properties may also include information such as pen location, pen pressure, pen angle, and the like.
In one embodiment, stylomouse 410 may map mouse and/or keyboard inputs to pen gestures than may be performed with a digital pen. A gesture is a pen movement or a combination of movements that are assigned special behavior within an application or OS in order to implement the behavior assigned to the gesture. Such pen gestures may include a tap, a double tap, a press and hold, a flick, and the like.
Embodiments of mappings between mouse/keyboard and a digital pen are shown in Table 1 below.
As shown in Table 1, a pen flick may be created using a combination of keyboard 404 and mouse 405. For example, the location of the cursor using mouse 405 defines the start point of the flick. A key from a number keypad on keyboard 404 defines the flick direction. For example, key “8” defines a flick up while key “6” defines a flick to the rig ht.
Pen actions engine 412 generates pen actions that may be used by pen input simulation system 414. These pen actions may be generated based at least in part on a type of digital pen (defined by pen properties) along with a stroke (defined by stroke properties). Pen actions may also include pen gestures.
Embodiments of pen actions engine 412 may describe properties of a particular type of digital pen. Pen properties may include physical pen dimensions, such as the physical height and width of the digital pen, and logical pen dimensions, such as the logical height and width of the digital pen in logical units used by the OS, such as pixels.
Pen properties may include the kinds of input data supported by the pen. Such input data may include positioning method (e.g., Cartesian or Polar). Other exemplary inputs include the range of pen pressure supported by the pen and the range of pen tilts supported by the pen.
Pen actions engine 412 may generate stroke properties. Such stroke properties may include stroke start point, stroke start time, stroke end point, and stroke end time. Stroke properties may include the interpoint/packet timing which describes the time between successive points that make up the stroke.
In one embodiment, stroke points, such as the start and end points, may be described using a Cartesian co-ordinate system defined by x-y positions. In another embodiment, the stroke points may be described using a Polar co-ordinate system that may be defined by (r, ⊖), where r is the radial distance from a reference origin and ⊖ is the angle in a counterclockwise direction of the point from the x-axis.
A stroke may have other properties depending on the properties supported by the writing surface on the tablet PC. A pen pressure property includes the starting pressure for the stroke and the pressure gradient of the stroke. A pen tilt property includes the starting pen tilt and the pen tilt changes during the stroke, as measured on each of the axes. A speed property describes the speed of the stroke. Stroke properties may also include the state of buttons on the digital pen. For example, the state of pen barrel button 512 may include pressed or un-pressed during the stroke.
In one embodiment, pen actions engine 412 may generate stroke properties that do not have corresponding input data from mouse 405 or keyboard 404. For example, pen pressure may not necessarily be inputted from a mouse movement. Pen actions engine 412 may generate the pen pressure property so that a pen pressure property may be provided to pen input simulation system 414. In one embodiment, the pen pressure property is provided by a programmable setting of pen actions engine 412. Other such generated stroke properties may include pen tilt and pen speed.
A stroke may include other properties that may be generated by pen actions engine 412 to describe a stroke. These additional properties may be used to describe a complex stroke such as a curved stroke and a composite stroke. A composite stroke is a stroke made up of more than one stroke in succession.
Turning to
In
Qbutton 610 captures real-time data from keyboard 604. Qbutton maps a key press at keyboard 604 to the properties of a tablet PC button. For example, the key “A” on keyboard 604 may be mapped to a particular tablet PC button, such as button 521 of tablet PC 502. Key combinations may also be mapped to a tablet PC button. For example, a Ctrl-A combination may map to a tablet PC button.
Qbutton 610 passes the button mapping to a button actions engine 612. Button actions engine 612 generates the properties associated with the button press of the corresponding tablet PC button. The tablet PC button data is then passed to button input simulation system 614 which in turn injects the tablet PC button data into OS device stack 220.
Using embodiments herein, software developers may test and market software for new hardware even though the new hardware is not yet available. Often times, new hardware develops at a slower rate than software to utilize the hardware. For example, 64-bit tablet PCs may not be available, yet software developers want to test their software on a 64-bit tablet PC using digital pen inputs. In another example, a tablet PC OS may support up to 32 tablet PC buttons, however, a tablet PC may not exist that has 32 buttons. Embodiments herein allow software developers to test software on a desktop computer and simulate the behavior of tablet input devices such as digital pens and tablet PC buttons using old input devices such as a mouse and a keyboard.
Various operations of embodiments of the present invention are described herein. In one embodiment, one or more of the operations described may constitute computer readable instructions stored on computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it will be understood that not all operations are necessarily present in each embodiment of the invention.
The above description of embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. While specific embodiments and examples of the invention are described herein for illustrative purposes, various equivalent modifications are possible, as those skilled in the relevant art will recognize in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the following claims are to be construed in accordance with established doctrines of claim interpretation.
