Patent Application
20090079692
The present invention relates generally to computing systems and, more particularly, to a method and system for enabling selection and use of objects in a computer system. It has been developed primarily to enable users to receive useful information from a paper-based user interface.
The following applications have been filed by the Applicant simultaneously with the present application:
The disclosures of these co-pending applications are incorporated herein by reference. The above applications have been identified by their filing docket number, which will be substituted with the corresponding application number, once assigned.
The following patents or patent applications filed by the applicant or assignee of the present invention are hereby incorporated by cross-reference.
The Applicant has previously described a method of enabling users to access information from a computer system via a printed substrate e.g. paper. The substrate has coded data printed thereon, which is read by an optical sensing device when the user interacts with the substrate using the sensing device. A computer receives interaction data from the sensing device and uses this data to determine what action is being requested by the user. For example, a user may select a printed hyperlink using a sensing device and retrieve a corresponding webpage via a display device or printer.
It would be desirable to enhance the functionality of the above-described system. Enhanced functionality would encourage greater use of the system and, hence, generate increased revenue streams for system providers.
More particularly, it would be desirable to provide users with the ability to complement web-based interactions with paper-based interactions.
In a first aspect the present invention provides a method of generating a clipping from a printed substrate, said substrate having a graphic image and a position-coding pattern disposed thereon, said position-coding pattern identifying an identity and a plurality of locations, said method comprising the steps of:
Optionally, said clipping identifies a clipping image, said clipping image comprising a clipping visual description describing the part of the graphic image contained in said clipping zone.
Optionally, the method further comprising the step of storing said clipping image on a clipping image store, wherein said stored clipping image is indexed with said clipping identity.
Optionally, the method further comprising displaying said clipping on a display device, said displaying comprising the steps of:
Optionally, the method further comprising displaying said clipping on a display device, said displaying comprising the steps of:
Optionally, said clipping zone is indicated by a path of the sensing device relative to the substrate.
Optionally, said clipping identity identifies a time at which said clipping is generated.
Optionally, said clipping zone contains at least one interactive element, such that said clipping is an interactive clipping.
Optionally, said at least one interactive element is selected from the group comprising: a hyperlink and an input field.
Optionally, said clipping identity identifies a clipping input description describing said at least one interactive element via said page description and said clipping zone.
Optionally, said clipping further identifies a clipping input service configured for interpreting an interaction with said clipping via a graphical user interface.
Optionally, said clipping further identifies citation information associated with the part of the graphic image contained in said clipping zone.
Optionally, the clipping is represented, at least partially, using an HTML fragment containing an image map.
Optionally, the image map is selected from the group comprising: a server-side image map and a client-side image map.
Optionally, said clipping is represented using a script.
Optionally, the method further comprising at least one of: pasting the clipping into an application, e-mailing the clipping, and pasting the clipping into a blog.
Optionally, the method further comprising the steps of:
Optionally, the interaction with the at least one interactive element identifies a click location and the clipping identity.
Optionally, the clipping identity and the click location are used to identify the corresponding page description and interpret the interaction.
Optionally, the method further comprising identifying a target URI and displaying a corresponding target resource on a display device.
In a second aspect the present invention provides a method of generating a self-contained interactive clipping from a printed substrate, said substrate having a graphic image and a position-coding pattern disposed thereon, said position-coding pattern identifying an identity and a plurality of locations, said method comprising the steps of:
Optionally, the method further comprising displaying said clipping on a display device.
Optionally, said clipping zone is indicated by a path of the sensing device relative to the substrate.
Optionally, said at least one interactive element is selected from the group comprising: a hyperlink and an input field.
Optionally, the clipping is represented, at least partially, using an HTML fragment containing an image map explicitly hyperlinked to at least one target URI.
Optionally, the image map is selected from the group comprising: a server-side image map and a client-side image map.
Optionally, said clipping is represented using a script.
Optionally, the method further comprising at least one of: pasting the clipping into an application, e-mailing the clipping, and pasting the clipping into a blog.
Optionally, the method further comprising the steps of:
Optionally, the interaction with the interactive element identifies a click location and a target URI using the clipping input description contained in the clipping.
Optionally, the method further comprising displaying a target resource corresponding to said target URI on said display device.
Preferred and other embodiments of the invention will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
The present invention is used in connection with the Applicant's netpage system, which has been described in detail in the cross-referenced patent applications identified above.
In brief summary, the preferred form of the netpage system employs a computer interface in the form of a mapped surface, that is, a physical surface which contains references to a map of the surface maintained in a computer system. The map references can be queried by an appropriate sensing device. Depending upon the specific implementation, the map references may be encoded visibly or invisibly, and defined in such a way that a local query on the mapped surface yields an unambiguous map reference both within the map and among different maps. The computer system can contain information about features on the mapped surface, and such information can be retrieved based on map references supplied by a sensing device used with the mapped surface. The information thus retrieved can take the form of actions which are initiated by the computer system on behalf of the operator in response to the operator's interaction with the surface features.
In its preferred form, the netpage system relies on the production of, and human interaction with, netpages. These are pages of text, graphics and images printed on ordinary paper, but which work like interactive webpages. Information is encoded on each page using ink which is substantially invisible to the unaided human eye. The ink, however, and thereby the coded data, can be sensed by an optically imaging sensing device (“reader”) and transmitted to the netpage system. The sensing device may take the form of, for example, a clicker (for clicking on a specific position on a surface), a pointer having a stylus (for pointing or gesturing on a surface using pointer strokes), or a pen having a marking nib (for marking a surface with ink when pointing, gesturing or writing on the surface). Any references herein to “pen” or “netpage pen” are provided by way of example only. It will, of course, be appreciated that the netpage pen may take the form of any suitable optically imaging sensing device or reader.
In one embodiment, active buttons and hyperlinks on each page can be clicked with the sensing device to request information from the network or to signal preferences to a network server. In one embodiment, text written by hand on a netpage is automatically recognized and converted to computer text in the netpage system, allowing forms to be filled in. In other embodiments, signatures recorded on a netpage are automatically verified, allowing e-commerce transactions to be securely authorized. In other embodiments, text on a netpage may be clicked or gestured to initiate a search based on keywords indicated by the user.
As illustrated in
The surface coding pattern 3 typically takes the form of a grid of target dots, which comprises a collection of tags 4. One such tag 4 is shown in the shaded region of
A corresponding page description 5, stored on the netpage network, describes the individual elements of the netpage. In particular it has an input description describing the type and spatial extent (zone) of each interactive element (i.e. text field or button in the example), to allow the netpage system to correctly interpret input via the netpage. The submit button 6, for example, has a zone 7 which corresponds to the spatial extent of the corresponding graphic 8.
As illustrated in
The relay device 601 performs the basic function of relaying interaction data to a page server 10, which interprets the interaction data. As shown in
The netpage printer 601b is able to deliver, periodically or on demand, personalized newspapers, magazines, catalogs, brochures and other publications, all printed at high quality as interactive netpages. Unlike a personal computer, the netpage printer is an appliance which can be, for example, wall-mounted adjacent to an area where the morning news is first consumed, such as in a user's kitchen, near a breakfast table, or near the household's point of departure for the day. It also comes in tabletop, desktop, portable and miniature versions. Netpages printed on-demand at their point of consumption combine the ease-of-use of paper with the timeliness and interactivity of an interactive medium.
Alternatively, the netpage relay device 601 may be a portable device, such as a mobile phone or PDA, a laptop or desktop computer, or an information appliance connected to a shared display, such as a TV. If the relay device 601 is not a netpage printer 601b which prints netpages digitally and on demand, the netpages may be printed by traditional analog printing presses, using such techniques as offset lithography, flexography, screen printing, relief printing and rotogravure, as well as by digital printing presses, using techniques such as drop-on-demand inkjet, continuous inkjet, dye transfer, and laser printing. As will be explained in more detail below, the present invention provides an alternative means which enables users conveniently to generate netpages, as well as scan or print images.
As shown in
The netpage relay device 601 can be configured to support any number of readers 400, and a reader can work with any number of netpage relays. In the preferred implementation, each netpage reader 400 has a unique identifier. This allows each user to maintain a distinct profile with respect to a netpage page server 10 or application server 13.
A netpage pen may be registered with a netpage registration server 11 and linked to one or more payment card accounts. This allows e-commerce payments to be securely authorized using the netpage pen. The netpage registration server compares the signature captured by the netpage pen with a previously registered signature, allowing it to authenticate the user's identity to an e-commerce server. Other biometrics can also be used to verify identity. One version of the netpage pen includes fingerprint scanning, verified in a similar way by the netpage registration server.
Netpages are the foundation on which a netpage network is built. They provide a paper-based user interface to published information and interactive services.
As shown in
Multiple netpages (for example, those printed by analog printing presses) can share the same page description. However, to allow input through otherwise identical pages to be distinguished, each netpage may be assigned a unique page identifier in the form of a page ID (or, more generally, an impression ID). The page ID has sufficient precision to distinguish between a very large number of netpages.
Each reference to the page description 5 is repeatedly encoded in the netpage pattern. Each tag (and/or a collection of contiguous tags) identifies the unique page on which it appears, and thereby indirectly identifies the page description 5. Each tag also identifies its own position on the page. Characteristics of the tags are described in more detail below.
Tags are typically printed in infrared-absorptive ink on any substrate which is infrared-reflective, such as ordinary paper, or in infrared fluorescing ink. Near-infrared wavelengths are invisible to the human eye but are easily sensed by a solid-state image sensor with an appropriate filter.
A tag is sensed by a 2D area image sensor in the netpage reader 400, and the tag data is transmitted to the netpage system via the nearest netpage relay device 601. The reader 400 is wireless and communicates with the netpage relay device 601 via a short-range radio link. It is important that the reader recognize the page ID and position on every interaction with the page, since the interaction is stateless. Tags are error-correctably encoded to make them partially tolerant to surface damage.
The netpage page server 10 maintains a unique page instance for each unique printed netpage, allowing it to maintain a distinct set of user-supplied values for input fields in the page description 5 for each printed netpage 1.
Each tag 4, contained in the position-coding pattern 3, identifies an absolute location of that tag within a region of a substrate.
Each interaction with a netpage should also provide a region identity together with the tag location. In a preferred embodiment, the region to which a tag refers coincides with an entire page, and the region ID is therefore synonymous with the page ID of the page on which the tag appears. In other embodiments, the region to which a tag refers can be an arbitrary subregion of a page or other surface. For example, it can coincide with the zone of an interactive element, in which case the region ID can directly identify the interactive element.
As described in some of the Applicant's previous applications (e.g. U.S. Pat. No. 6,832,717), the region identity may be encoded discretely in each tag 4. As described other of the Applicant's applications (e.g. U.S. application Ser. Nos. 12/025,746 & 12/025,765 filed on Feb. 5, 2008), the region identity may be encoded by a plurality of contiguous tags in such a way that every interaction with the substrate still identifies the region identity, even if a whole tag is not in the field of view of the sensing device.
Each tag 4 should preferably identify an orientation of the tag relative to the substrate on which the tag is printed. Strictly speaking, each tag 4 identifies an orientation of tag data relative to a grid containing the tag data. However, since the grid is typically oriented in alignment with the substrate, then orientation data read from a tag enables the rotation (yaw) of the netpage reader 400 relative to the grid, and thereby the substrate, to be determined.
A tag 4 may also encode one or more flags which relate to the region as a whole or to an individual tag. One or more flag bits may, for example, signal a netpage reader 400 to provide feedback indicative of a function associated with the immediate area of the tag, without the reader having to refer to a corresponding page description 5 for the region. A netpage reader may, for example, illuminate an “active area” LED when positioned in the zone of a hyperlink.
A tag 4 may also encode a digital signature or a fragment thereof. Tags encoding digital signatures (or a part thereof) are useful in applications where it is required to verify a product's authenticity. Such applications are described in, for example, US Publication No. 2007/0108285, the contents of which is herein incorporated by reference. The digital signature may be encoded in such a way that it can be retrieved from every interaction with the substrate. Alternatively, the digital signature may be encoded in such a way that it can be assembled from a random or partial scan of the substrate.
It will, of course, be appreciated that other types of information (e.g. tag size etc) may also be encoded into each tag or a plurality of tags.
For a full description of netpage tags 4, reference is made to the Applicant's previously filed patent applications identified above, the contents of which are herein incorporated by reference (see, for example, U.S. Pat. No. 6,832,717 and U.S. application Ser. Nos. 12/025,746 & 12/025,765 filed on Feb. 5, 2008, the contents of which are herein incorporated by reference).
A preferred embodiment of a document and page description class diagram is shown in
In the netpage system a document is described at three levels. At the most abstract level the document 836 has a hierarchical structure whose terminal elements 839 are associated with content objects 840 such as text objects, text style objects, image objects, etc. Once the document is printed on a printer with a particular page size, the document is paginated and otherwise formatted. Formatted terminal elements 835 will in some cases be associated with content objects which are different from those associated with their corresponding terminal elements, particularly where the content objects are style-related. Each printed instance of a document and page is also described separately, to allow input captured through a particular page instance 830 to be recorded separately from input captured through other instances of the same page description.
The presence of the most abstract document description on the page server allows a a copy of a document to be printed without being forced to accept the source document's specific format. The user or a printing press may be requesting a copy for a printer with a different page size, for example. Conversely, the presence of the formatted document description on the page server allows the page server to efficiently interpret user actions on a particular printed page.
A formatted document 834 consists of a set of formatted page descriptions 5, each of which consists of a set of formatted terminal elements 835. Each formatted element has a spatial extent or zone 58 on the page. This defines the active area of input elements such as hyperlinks and input fields.
A document instance 831 corresponds to a formatted document 834. It consists of a set of page instances 830, each of which corresponds to a page description 5 of the formatted document. Each page instance 830 describes a single unique printed netpage 1, and records the page ID 50 of the netpage. A page instance is not part of a document instance if it represents a copy of a page requested in isolation.
A page instance consists of a set of terminal element instances 832. An element instance only exists if it records instance-specific information. Thus, a hyperlink instance exists for a hyperlink element because it records a transaction ID 55 which is specific to the page instance, and a field instance exists for a field element because it records input specific to the page instance. An element instance does not exist, however, for static elements such as textflows.
A terminal element 839 can be a visual element or an input element. A visual element can be a static element 843 or a dyamnic element 846. An input element can be a hyperlink element 844 or a field element 845, as shown in
A page instance has a background field 833 which is used to record any digital ink captured on the page which does not apply to a specific input element.
In the preferred form of the invention, a tag map 811 is associated with each page instance to allow tags on the page to be translated into locations on the page.
A simple netpage network may consist of a distributed set of netpage page servers 10, netpage registration servers 11, netpage ID servers 12, netpage application servers 13, and netpage relay devices 601 connected via a network 19 such as the Internet, as shown in
The netpage registration server 11 is a server which records relationships between users, pens, printers and/or applications, and thereby authorizes various network activities. It authenticates users and acts as a signing proxy on behalf of authenticated users in application transactions. It also provides handwriting recognition services. As described above, a netpage page server 10 maintains persistent information about page descriptions and page instances. The netpage network includes any number of page servers, each handling a subset of page instances. Since a page server also maintains user input values for each page instance, clients such as netpage relays 601 send netpage input directly to the appropriate page server. The page server interprets any such input relative to the description of the corresponding page.
A netpage ID server 12 allocates document IDs 51 on demand, and provides load-balancing of page servers via its ID allocation scheme.
A netpage relay 601 uses the Internet Distributed Name System (DNS), or similar, to resolve a netpage page ID 50 into the network address of the netpage page server handling the corresponding page instance.
A netpage application server 13 is a server which hosts interactive netpage applications.
Netpage servers can be hosted on a variety of network server platforms from manufacturers such as IBM, Hewlett-Packard, and Sun. Multiple netpage servers can run concurrently on a single host, and a single server can be distributed over a number of hosts. Some or all of the functionality provided by netpage servers, and in particular the functionality provided by the ID server and the page server, can also be provided directly in a netpage appliance such as a netpage printer, in a computer workstation, or on a local network.
The active sensing device (or reader 400) of the netpage system may take the form of, for example, a clicker (for clicking on a specific position on a surface), a pointer having a stylus (for pointing or gesturing on a surface using pointer strokes), or a pen having a marking nib (for marking a surface with ink when pointing, gesturing or writing on the surface). A pen 101 is described herein, although it will be appreciated that clickers and pointers may be of similar construction. The pen 101 uses its embedded controller 134 to capture and decode netpage tags from a page via an image sensor. The image sensor is a solid-state device provided with an appropriate filter to permit sensing at only near-infrared wavelengths. As described in more detail below, the system is able to sense when the nib is in contact with the surface, and the pen is able to sense tags at a sufficient rate to capture human handwriting (i.e. at 200 dpi or greater and 100 Hz or faster). Information captured by the pen may be encrypted and wirelessly transmitted to the printer (or base station), the printer or base station interpreting the data with respect to the (known) page structure.
One embodiment of the netpage pen 101 operates both as a normal marking ink pen and as a non-marking stylus (i.e. as a pointer). The marking aspect, however, is not necessary for using the netpage system as a browsing system, such as when it is used as an Internet interface. Each netpage pen is registered with the netpage system and has a unique pen ID 61.
When the nib is in contact with a netpage, the pen determines its position and orientation relative to the page. The nib is attached to a force sensor, and the force on the nib is interpreted relative to a threshold to indicate whether the pen is “up” or “down”. This allows a interactive element on the page to be ‘clicked’ by pressing with the pen nib, in order to request, say, information from a network. Furthermore, the force is captured as a continuous value to allow, say, the full dynamics of a signature to be verified.
The pen determines the position and orientation of its nib on the netpage by imaging, in the infrared spectrum, an area 193 of the page in the vicinity of the nib. It decodes the nearest tag and computes the position of the nib relative to the tag from the observed perspective distortion on the imaged tag and the known geometry of the pen optics i.e. the spatial relationship between the nib and the optical axis. Although the position resolution of the tag may be low, because the tag density on the page is inversely proportional to the tag size, the adjusted position resolution is quite high, exceeding the minimum resolution required for accurate handwriting recognition.
Pen actions relative to a netpage are captured as a series of strokes. A stroke consists of a sequence of time-stamped pen positions on the page, initiated by a pen-down event and completed by the subsequent pen-up event. A stroke is also tagged with the page ID 50 of the netpage whenever the page ID changes, which, under normal circumstances, is at the commencement of the stroke.
Each netpage pen has a current selection 826 associated with it, allowing the user to perform copy and paste operations etc. The selection is timestamped to allow the system to discard it after a defined time period. The current selection describes a region of a page instance. It consists of the most recent digital ink stroke captured through the pen relative to the background area of the page. It is interpreted in an application-specific manner once it is submitted to an application via a selection hyperlink activation.
Each pen has a current nib 824. This is the nib last notified by the pen to the system. In the case of the default netpage pen described above, either the marking black ink nib or the non-marking stylus nib is current. Each pen also has a current nib style 825. This is the nib style last associated with the pen by an application, e.g. in response to the user selecting a color from a palette. The default nib style is the nib style associated with the current nib. Strokes captured through a pen are tagged with the current nib style. When the strokes are subsequently reproduced, they are reproduced in the nib style with which they are tagged.
The pen 101 may have one or more buttons 209, which are pressed by the user to select a mode of the pen. The button(s) are used to determine a behavior of the pen, which, in turn, determines how a stroke is interpreted by the page server 10. For example, a ‘clipping mode’ may be selected using a button 209 (see Section 7.3).
Whenever the pen is within range of a relay device 601 with which it can communicate, the pen slowly flashes its “online” LED. When the pen fails to decode a stroke relative to the page, it momentarily activates its “error” LED. When the pen succeeds in decoding a stroke relative to the page, it momentarily activates its “ok” LED.
A sequence of captured strokes is referred to as digital ink. Digital ink forms the basis for the digital exchange of drawings and handwriting, for online recognition of handwriting, and for online verification of signatures.
The pen is wireless and transmits digital ink to the relay device 601 via a short-range radio link. The transmitted digital ink is encrypted for privacy and security and packetized for efficient transmission, but is always flushed on a pen-up event to ensure timely handling in the printer.
When the pen is out-of-range of a relay device 601 it buffers digital ink in internal memory, which has a capacity of over ten minutes of continuous handwriting. When the pen is once again within range of a relay device, it transfers any buffered digital ink.
A pen can be registered with any number of relay devices, but because all state data resides in netpages both on paper and on the network, it is largely immaterial which relay device a pen is communicating with at any particular time.
A more detailed description of the Netpage pen may be found in the Applicant's earlier cross-referenced patents and patent applications (see, for example, U.S. Pat. No. 7,105,753; U.S. Pat. No. 7,015,901; U.S. Pat. No. 7,091,960; and US Publication No. 2006/0028459, the contents of which are herein incorporated by reference).
Referring to
Also mounted within the cover 107 is a tri-color status LED 116 on a flex PCB 117. The antenna 112 is also mounted on the flex PCB 117. The status LED 116 is mounted at the top of the pen 101 for good all-around visibility.
The pen described herein can operate both as a normal marking ink pen and as a non-marking stylus (‘pointer’). An ink pen cartridge 118 with nib 119 and a stylus 120 with stylus nib 121 are mounted side by side within the housing 102. Either the ink cartridge nib 119 or the stylus nib 121 can be brought forward through open end 122 of the metal end piece 114, by rotation of the pen top 105. Respective slider blocks 123 and 124 are mounted to the ink cartridge 118 and stylus 120, respectively. A rotatable cam barrel 125 is secured to the pen top 105 in operation and arranged to rotate therewith. The cam barrel 125 includes a cam 126 in the form of a slot within the walls 181 of the cam barrel. Cam followers 127 and 128 projecting from slider blocks 123 and 124 fit within the cam slot 126. On rotation of the cam barrel 125, the slider blocks 123 or 124 move relative to each other to project either the pen nib 119 or stylus nib 121 out through the hole 122 in the metal end piece 114. The pen 101 has three states of operation. By turning the top 105 through 90° steps, the three states are:
A second flex PCB 129, is mounted on an electronics chassis 130 which sits within the housing 102. The second flex PCB 129 mounts an infrared LED 131 for providing infrared radiation for projection onto the surface. An image sensor 132 is provided mounted on the second flex PCB 129 for receiving reflected radiation from the surface. The second flex PCB 129 also mounts a radio frequency chip 133, which includes an RF transmitter and RF receiver, and a controller chip 134 for controlling operation of the pen 101. An optics block 135 (formed from moulded clear plastics) sits within the cover 107 and projects an infrared beam onto the surface and receives images onto the image sensor 132. Power supply wires 136 connect the components on the second flex PCB 129 to battery contacts 137 which are mounted within the cam barrel 125. A terminal 138 connects to the battery contacts 137 and the cam barrel 125. A three volt rechargeable battery 139 sits within the cam barrel 125 in contact with the battery contacts. An contacts. An induction charging coil 140 is mounted about the second flex PCB 129 to enable recharging of the battery 139 via induction. The second flex PCB 129 also mounts an infrared LED 143 and infrared photodiode 144 for detecting displacement in the cam barrel 125 when either the stylus 120 or the ink cartridge 118 is used for writing, in order to enable a determination of the force being applied to the surface by the pen nib 119 or stylus nib 121. The IR photodiode 144 detects light from the IR LED 143 via reflectors (not shown) mounted on the slider blocks 123 and 124.
Rubber grip pads 141 and 142 are provided towards the end 108 of the housing 102 to assist gripping the pen 101, and top 105 also includes a clip 142 for clipping the pen 101 to a pocket.
The pen 101 is arranged to determine the position of its nib (stylus nib 121 or ink cartridge nib 119) by imaging, in the infrared spectrum, an area of the surface in the vicinity of the nib. It records the location data from the nearest location tag, and is arranged to calculate the distance of the nib 121 or 119 from the location tab utilising optics 135 and controller chip 134. The controller chip 134 calculates the orientation of the pen and the nib-to-tag distance from the perspective distortion observed on the imaged tag.
Utilising the RF chip 133 and antenna 112 the pen 101 can transmit the digital ink data (which is encrypted for security and packaged for efficient transmission) to the computing system.
When the pen is in range of a relay device 601, the digital ink data is transmitted as it is formed. When the pen 101 moves out of range, digital ink data is buffered within the pen 101 (the pen 101 circuitry includes a buffer arranged to store digital ink data for approximately 12 minutes of the pen motion on the surface) and can be transmitted later.
The nib ID allows the computing system to identify which nib (stylus nib 121 or ink cartridge nib 119) is presently being used. The computing system can vary its operation depending upon which nib is being used. For example, if the ink cartridge nib 119 is being used the computing system may defer producing feedback output because immediate feedback is provided by the ink markings made on the surface. Where the stylus nib 121 is being used, the computing system may produce immediate feedback output.
Since a user may change the nib 119, 121 between one stroke and the next, the pen 101 optionally records a nib ID for a stroke 175. This becomes the nib ID implicitly associated with later strokes.
Cartridges having particular nib characteristics may be interchangeable in the pen. The pen controller 134 may interrogate a cartridge to obtain the nib ID 175 of the cartridge. The nib ID 175 may be stored in a ROM or a barcode on the cartridge. The controller 134 notifies the system of the nib ID whenever it changes. The system is thereby able to determine the characteristics of the nib used to produce a stroke 175, and is thereby subsequently able to reproduce the characteristics of the stroke itself.
The controller chip 134 is mounted on the second flex PCB 129 in the pen 101.
Referring to
An image sensor interface 152 interfaces with the image sensor 132. A transceiver controller 153 and base band circuit 154 are also included to interface with the RF chip 133 which includes an RF circuit 155 and RF resonators and inductors 156 connected to the antenna 112.
The controlling processor 145 captures and decodes location data from tags from the surface via the image sensor 132, monitors the force sensor photodiode 144, controls the LEDs 116, 131 and 143, and handles short-range radio communication via the radio transceiver 153. It is a medium-performance (˜40 MHz) general-purpose RISC processor. The controlling processor 145 also receives input via mode selection button(s) 209, which can be used to control interpretation of pen-substrate interactions.
The processor 145, digital transceiver components (transceiver controller 153 and baseband circuit 154), image sensor interface 152, flash memory 147 and 512 KB DRAM 148 are integrated in a single controller ASIC. Analog RF components (RF circuit 155 and RF resonators and inductors 156) are provided in the separate RF chip.
The image sensor 132 may be a 215×215 pixel CCD (such a sensor is produced by Matsushita Electronic Corporation, and is described in a paper by Itakura, K T Nobusada, N Okusenya, R Nagayoshi, and M Ozaki, “A 1 mm 50 k-Pixel IT CCD Image Sensor for Miniature Camera System”, IEEE Transactions on Electronic Devices, Volt 47, number 1, January 2000, which is incorporated herein by reference) with an IR filter.
The controller ASIC 134 enters a quiescent state after a period of inactivity when the pen 101 is not in contact with a surface. It incorporates a dedicated circuit 150 which monitors the force sensor photodiode 144 and wakes up the controller 134 via the power manager 151 on a pen-down event.
The radio transceiver communicates in the unlicensed 900 MHz band normally used by cordless telephones, or alternatively in the unlicensed 2.4 GHz industrial, scientific and medical (ISM) band, and uses frequency hopping and collision detection to provide interference-free communication.
In an alternative embodiment, the pen incorporates an Infrared Data Association (IrDA) interface for short-range communication with a base station or netpage printer.
In a further embodiment, the pen 101 includes a pair of orthogonal accelerometers mounted in the normal plane of the pen 101 axis. The accelerometers 190 are shown in
The provision of the accelerometers enables this embodiment of the pen 101 to sense motion without reference to surface location tags, allowing the location tags to be sampled at a lower rate. Each location tag ID can then identify an object of interest rather than a position on the surface. For example, if the object is a user interface input element (e.g. a command button), then the tag ID of each location tag within the area of the input element can directly identify the input element.
The acceleration measured by the accelerometers in each of the x and y directions is integrated with respect to time to produce an instantaneous velocity and position.
Since the starting position of the stroke is not known, only relative positions within a stroke are calculated. Although position integration accumulates errors in the sensed acceleration, accelerometers typically have high resolution, and the time duration of a stroke, over which errors accumulate, is short.
The netpage relay device 601 receives data relating to a stroke from the pen 101 when the pen is used to interact with a netpage 1. The coded data 3 of the tags 4 is read by the pen when it is used to execute a movement, such as a stroke. The data allows the identity of the particular page to be determined and an indication of the positioning of the pen relative to the page to be obtained. Interaction data comprising the page ID 50 and at least one position of the pen, is transmitted to the relay device 601, where it resolves, via the DNS, the page ID 50 of the stroke into the network address of the netpage page server 10 which maintains the corresponding page instance 830. In one form of the netpage system, the relay device 601 then transmits the stroke to the page server. If the page was recently identified in an earlier stroke, then the relay device may already have the address of the relevant page server in its cache. Each netpage consists of a compact page layout maintained persistently by a netpage page server (see below). The page layout refers to objects such as images, fonts and pieces of text, typically stored elsewhere on the netpage network.
When the page server 10 receives the stroke from the pen, it retrieves the page description to which the stroke applies, and determines which element of the page description the stroke intersects. It is then able to interpret the stroke in the context of the type of the relevant element.
A “click” is a stroke where the distance and time between the pen down position and the subsequent pen up position are both less than some small maximum. An object which is activated by a click typically requires a click to be activated, and accordingly, a longer stroke is ignored. The failure of a pen action, such as a “sloppy” click, to register may be indicated by the lack of response from the pen's “ok” LED.
There are two kinds of input elements in a netpage page description: hyperlinks and form fields. Input through a form field can also trigger the activation of an associated hyperlink.
A hyperlink is a means of sending a message to a remote application, and typically elicits a displayed or printed response in the netpage system.
A hyperlink element 844 identifies the application 71 which handles activation of the hyperlink, a link ID 54 which identifies the hyperlink to the application, an “alias required” flag which asks the system to include the user's application alias ID 65 in the hyperlink activation, and a description which is used when the hyperlink is recorded as a favorite or appears in the user's history. The hyperlink element class diagram is shown in
When a hyperlink is activated, the page server sends a request to an application somewhere on the network. The application is identified by an application ID 64, and the application ID is resolved in the normal way via the DNS. There are three types of hyperlinks: general hyperlinks 863, form hyperlinks 865, and selection hyperlinks 864, as shown in
The corresponding hyperlink instance 862 records a transaction ID 55 which can be specific to the page instance on which the hyperlink instance appears. The transaction ID can identify user-specific data to the application, for example a “shopping cart” of pending purchases maintained by a purchasing application on behalf of the user.
The system includes the pen's current selection 826 in a selection hyperlink activation. The system includes the content of the associated form instance 868 in a form hyperlink activation, although if the hyperlink has its “submit delta” attribute set, only input since the last form submission is included. The system includes an effective return path in all hyperlink activations.
A hyperlinked group 866 is a group element 838 which has an associated hyperlink, as shown in
A form defines a collection of related input fields used to capture a related set of inputs through a printed netpage. A form allows a user to submit one or more parameters to an application software program running on a server.
A form 867 is a group element 838 in the document hierarchy. It ultimately contains a set of terminal field elements 839. A form instance 868 represents a printed instance of a form. It consists of a set of field instances 870 which correspond to the field elements 845 of the form. Each field instance has an associated value 871, whose type depends on the type of the corresponding field element. Each field value records input through a particular printed form instance, i.e. through one or more printed netpages. The form class diagram is shown in
Each form instance has a status 872 which indicates whether the form is active, frozen, submitted, void or expired. A form is active when first printed. A form becomes frozen once it is signed or once its freeze time is reached. A form becomes submitted once one of its submission hyperlinks has been activated, unless the hyperlink has its “submit delta” attribute set. A form becomes void when the user invokes a void form, reset form or duplicate form page command. A form expires when its specified expiry time is reached, i.e. when the time the form has been active exceeds the form's specified lifetime. While the form is active, form input is allowed. Input through a form which is not active is instead captured in the background field 833 of the relevant page instance. When the form is active or frozen, form submission is allowed. Any attempt to submit a form when the form is not active or frozen is rejected, and instead elicits an form status report.
Each form instance is associated (at 59) with any form instances derived from it, thus providing a version history. This allows all but the latest version of a form in a particular time period to be excluded from a search.
All input is captured as digital ink. Digital ink 873 consists of a set of timestamped stroke groups 874, each of which consists of a set of styled strokes 875. Each stroke consists of a set of timestamped pen positions 876, each of which also includes pen orientation and nib force. The digital ink class diagram is shown in
A field element 845 can be a checkbox field 877, a text field 878, a drawing field 879, or a signature field 880. The field element class diagram is shown in
A checkbox field has an associated boolean value 881, as shown in
A text field has an associated text value 882, as shown in
A signature field has an associated digital signature value 883, as shown in
A field element is hidden if its “hidden” attribute is set. A hidden field element does not have an input zone on a page and does not accept input. It can have an associated field value which is included in the form data when the form containing the field is submitted.
“Editing” commands, such as strike-throughs indicating deletion, can also be recognized in form fields.
Because the handwriting recognition algorithm works “online” (i.e. with access to the dynamics of the pen movement), rather than “offline” (i.e. with access only to a bitmap of pen markings), it can recognize run-on discretely-written characters with relatively high accuracy, without a writer-dependent training phase. A writer-dependent model of handwriting is automatically generated over time, however, and can be generated up-front if necessary,
Digital ink, as already stated, consists of a sequence of strokes. Any stroke which starts in a particular element's zone is appended to that element's digital ink stream, ready for interpretation. Any stroke not appended to an object's digital ink stream is appended to the background field's digital ink stream.
Digital ink captured in the background field is interpreted as a selection gesture. Circumscription of one or more objects is generally interpreted as a selection of the circumscribed objects, although the actual interpretation is application-specific.
Table 1 summarises some typical pen interactions with a netpage.
The system maintains a current selection for each pen. The selection consists simply of the most recent stroke captured in the background field. The selection is cleared after an inactivity timeout to ensure predictable behavior.
The raw digital ink captured in every field is retained on the netpage page server and is optionally transmitted with the form data when the form is submitted to the application. This allows the application to interrogate the raw digital ink should it suspect the original conversion, such as the conversion of handwritten text. This can, for example, involve human intervention at the application level for forms which fail certain application-specific consistency checks. As an extension to this, the entire background area of a form can be designated as a drawing field. The application can then decide, on the basis of the presence of digital ink outside the explicit fields of the form, to route the form to a human operator, on the assumption that the user may have indicated amendments to the filled-in fields outside of those fields.
The netpage system provides a mechanism to allow users to select an object on a printed netpage and submit it to an application, e.g. to associate the selected object with another object in the netpage system.
In one preferred embodiment, the selection mechanism is effected by circumscribing the graphical representation of an object using the netpage pen. The user may then submit the selected object to an application by activating a selection hyperlink via the same or another printed netpage.
The registration server maintains a current selection for each pen, describing a region of a page instance from which the selection has been made. This description includes the most recent digital ink stroke captured by way of transmitted signals from the pen relative to the background area of the page. Pen strokes are interpreted in an application-specific manner once they are submitted to an application via a selection hyperlink activation. When the user ‘clicks’ on a selection hyperlink, the page server obtains the pen's current selection from the registration server and transmits it to the corresponding application as part of the selection hyperlink activation, thus associating the two objects.
When the application receives a selection hyperlink activation, it retrieves the content of the selection from the page server which manages the page from which the selection was made. The application may then retrieve the selection as formatted data, allowing it to interpret the object in the form of the selected region in an application-specific manner in relation to the formatted data, or as unformatted data, allowing it to rely on the page server to interpret the selection region in a meaningful way.
When requesting unformatted data, the application may specify a desired scope to assist the interpretation of the selected region by the page server. Possible scopes include letter, word, phrase, line, paragraph and article. If the page server is unable to interpret the selection region region according to the desired scope, it may reject the application's request for the content of the selection.
The selection content returned by the page server to the application may include field values. Typically however, only field values which have been submitted as part of a form submission are included.
An author of a document can assist selection of articles by grouping all the elements of an article into an article group 507, as shown in
The protocol for selection hyperlink activation and subsequent selection content retrieval, illustrated diagrammatically in
When page server A receives a selection hyperlink ‘click’, it retrieves 510 the current selection associated with the pen from the registration server. The selection is described by a page ID 511 and a region 512. The page server then constructs a selection hyperlink request 508 (i.e. a specialization of the hyperlink request 934, shown in
As an alternative, page server A could sign the selection sent to the application, allowing page server B to easily verify that the selection supplied by the application has not been modified. Once page server B has the selection, it retrieves the selected page from its database and determines the content of the selection according to the application's desired format and scope. Finally the page server returns the selection content 515 to the application for application-specific processing.
If the page on which the selection was made was generated by the same application as that handling the selection hyperlink activation, then the application has direct access to the page (i.e. the application can retrieve the entire page from the relevant page server by way of the page ID of the page) and may already have done so before the selection hyperlink is activated. In this case, the application preferably interprets the selection region without reference to the selection content retrieval mechanism.
Selection of an object may of course alternatively be performed using a draggable command. This has an advantage in that a selection stroke can be distinguished from a normal input stroke wherever the stroke is made. The user may therefore select an object without inadvertently entering an input stroke into a field. As such, a user is also able to select an object that resides entirely within a field relatively easily.
The application may define the selection command in the form of a selection page server command 509, as shown in
When a selection page server command is activated by a user, the page server forwards the corresponding stroke to the registration server to be recorded as the current selection for the pen. Apart from this difference, the selection mechanism operates in the same way as previously described.
As an example, in a netpage e-mail application, clicking on an <attach> button at the bottom of every e-mail composition page effects attachment of the current selection at the current end of the body of the e-mail. The attachment can consist of any object or objects, which are capable of selection, on any netpage page. The user may have made this selection from, say, selecting text (eg. word/sentence/paragraph/article) from another netpage page, or selecting a photograph (eg. by circumscription) from a photo collection page. The entire e-mail may then be reprinted with the attachment included, additional pages being automatically added to the e-mail to accommodate the attachment.
A formatted element 835 associated with a textflow element 848 is a formatted textflow fragment 1018. The formatted textflow fragment consists of a set of formatted text lines 1020. Each formatted text line has a spatial extent or zone on the page. Each formatted text line consists of a set of formatted word fragments 1022, each of which has its own zone. The zone of formatted text line is the union of the zones of its word fragments. Each formatted word fragment is associated with a styled word 1014. Where a styled word is broken across multiple lines it has multiple formatted word fragments. Where the entire styled word lies within a single line it has a single formatted word fragment. Each formatted word fragment consists of a set of formatted characters 1024, each of which has a zone and is associated with a styled character. The zone of a formatted word fragment is the union of the zones of its formatted characters.
The user can utilize a number of different pen gestures to effect text selection. A click on a word can be used to select that word. The position of the click is compared with the zone of each word, and the word whose zone it lies within is selected. Circumscription (or lassoing) can be used to select of one or more words. The degree of overlap between the region enclosed by the circumscribing stroke and the zone of each word is used to determine whether the word is selected. The required overlap can be configured by the user. Underlining can be used to select one or more words. A line-like stroke which doesn't otherwise select any words is interpreted as an underline. Since the orientation of the text is known, the text which lies above the underlining stroke can be identified.
As described previously, the current selection is available to an application which is the target of a selection hyperlink 864. When an application receives a selection hyperlink activation, it can request the current selection.
A number of useful applications can utilize text selections. The selected text can be copied to the clipboard of the user's graphical user interface (GUI) operating system (e.g. Microsoft Windows, Apple Macintosh OS X, or Linux) for use by other GUI applications.
In addition to utilizing an explicit <copy text> command, the user can operate in a mode where all strokes, if interpretable as selection strokes, cause the selected text to be automatically copied to the clipboard of the GUI operating system. The user can also cause such copying to occur by utilizing one or more specific gestures, such as tapping on a word twice, or circumscribing a set of words twice, or writing the letter “C” and then circling it.
Each command button 1038 through 1046 is implemented as a selection hyperlink.
When user invokes a selection hyperlink which causes the target application to request the currently selected text, it is possible for the most recent strokes entered by the user to lie in a text field. As a configurable option (and as a per-request option) the system delivers the corresponding recognized text in place of any previously selected text.
The entire collection of command buttons shown in
As described earlier in relation to the <copy text> command, the user can also invoke these commands by utilizing a specific gestures, such as writing a corresponding letter and then circling it, e.g. “C” for <copy>, “S” for <search>, “D” for <dictionary>, “E” for <encyclopedia>, and “T” for <translate>.
An application which is a target of a selection hyperlink can query the content of the most recent selection, and can specify the format that the selection should be returned in. For example, the application can use the selection to retrieve text (a word, multiple words, a line, multiple lines, a paragraph, an article, etc.), an object (such as any of the object types listed in
The Netpage system also allows a user to take a virtual clipping from a printed page. A virtual clipping appears visually as a fragment of the printed page, bounded by the hand-drawn stroke used to circumscribe the clipping. The clipping is copied to the clipboard of the operating system of the user's display device, e.g. the user's PC, PDA or mobile phone. The user can subsequently paste the clipping into another application or document, such as a word processing document, an e-mail, or a blog. A clipping may include, by default, any digital ink strokes on the original page.
The application may also request a “clipping”, i.e. graphic page content clipped to the most recent selection stroke. A clipping optionally remains active, i.e. hyperlinks and/or fields embedded in the page become part of the clipping, e.g. encoded in HTML (see Section 7.4). A clipping includes, by default, any digital ink strokes on the original page.
Using a non-marking Netpage pointer the user can take a clipping using a single clipping stroke. Using a marking Netpage pen the user can identify a stroke as a clipping stroke via a subsequent action, such as clicking on a ‘copy’ command on command card or selecting a ‘copy’ command from the option menu.
Many other commands which copy objects to the clipboard of the GUI operating system are possible. It is also possible to provide a generic <copy> command which causes the current selection to be advertised on the clipboard in multiple formats (e.g. plain text, rich text, HTML, image, document, or document reference). The GUI application which retrieves the selection from the clipboard can then allow the user to select the desired format. Most GUI operating systems allow a form of “lazy” copying to the clipboard, where the copying application initially only copies an object reference to the clipboard, and only copies the object to the clipboard in its final format once it is notified that an application has attempted to retrieve the object from the clipboard.
An interactive clipping retains the interactivity of the original printed page. Any user interaction with the clipping is interpreted in much the same way as a corresponding user interaction with the original printed page. At a minimum an interactive clipping supports explicit and implicit hyperlinks. Depending on implementation, it may also support gestural commands and the context-sensitive option menu.
An interactive clipping provides the user with greater utility than a passive clipping, and allows the publisher of the original page to obtain value from user interactions with the clipping. A printed Netpage publication is an extension to the publisher's web presence, and each interactive clipping further extends this web presence.
As shown in
In order to minimize the size of the clipping 310, the clipping image 312 is stored separately and may be shared by a number of clippings. The clipping image 312 may be stored explicitly, or it may be generated from the original impression on demand. For the purpose of generating the clipping image 312 from the impression, the clipping information needs to record the entire clipping path rather than just the offset between the clipping and the impression, as well as the time the clipping took place to allow digital ink (etc.) added to the impression after the clipping took place to be ignored.
The interactive clipping is represented using an HTML fragment containing a server-side image map (see W3C, Server-Side Image Maps, HTML 4.01 Specification, W3C Recommendation 24 Dec. 1999, http://www.w3.org/TR/html1401/struct/objects.html#h-13.6.2):
Using HTML allows the interactive clipping to be supported by popular word processors, e-mail programs and blogs, as well as the clipboards of popular operating systems.
For illustrative purposes in the example the clipping ID is 1234. The anchor element identifies the clipping input service, and the id parameter in the input service URI identifies the clipping. The image element identifies the clipping image and the fact that it's an image map.
A click on the clipping image generates a URI of the following form:
For illustrative purposes in the example the click coordinates are (25,37). Note that the syntax of the parameters in the URI is not strictly conformant, since HTML does not define a way to combine image-map ‘query’ parameters with other query parameters. However, the indicated syntax is generally supported.
When displaying a clipping, the client uses a clipping image URI to retrieve the corresponding clipping image 312 from the Netpage clipping image web service 332. The image service 332 may in turn rely on the Netpage document service 336 to render the clipping image from the original impression. In this case the image service 332 extracts the clipping ID from the clipping image URI and passes it to the Netpage document service 336.
When handling a click on the clipping image 312, the client 330 generates a URI for the Netpage clipping input web service 334, including the clipping ID and click location. The input service 334 in turn relies on the Netpage document service 336 to identify the impression from the clipping ID, and interpret the click location relative to the impression (suitably adjusted according to the offset between the impression origin and clipping origin). The Netpage document service 336 returns the URI of the final target of the click, and the input service 334 returns this to the client as a redirect. The client uses the target URI to retrieve and display the target web page (or other web resource) from the target web service 338.
Although the Netpage image web service 332, input web service 334 and document service 336 are shown here as separate services, they may be implemented as multiple interfaces to a single service (e.g. the page server 10).
Server-side image maps provide maximum flexibility, since they allow the exact click location to be interpreted against the original impression's page description. However, some users object to server-side image maps because they don't reveal a click's target URI. Client-side image maps may be used in place of server-side image maps with the added complication that implicit hyperlinks, e.g. associated with images and text, must be made explicit in the HTML of the interactive clipping.
An impression may bear digital ink and other attached objects (e.g. photos). When the user takes a clipping it can be automatically placed on the clipboard in multiple formats, including variants that include and exclude digital ink (and other objects). Variants can also be included that are passive rather than interactive. The user can then use the ‘paste special’ function provided in most applications to select an appropriate variant of the clipping.
An image-map only supports clicks. To support more complex gestures, such as swipe to search or access to a contextual option menu, an interactive clipping can be represented using a script (i.e. widget-style). However, this makes it less compatible with extant HTML clients, either due to their implementation or their security policies.
Where the citation for the clipping is known, this should be displayed below the clipping in the form of an anchor linking to the publication's web presence, as shown in
For non-graphical HTML clients it is important for the user to have a way of viewing and interacting with the clipping. This can be provided via a separate anchor, with a distinct URI, included after the image map anchor. This is preferable to using the alternate text attribute of the image-map image element, since that generates an image-map click with a location of (0,0).
The non-graphical anchor should point to a web page that displays the clipping and allows interactions with it. Ideally this web page will also support script-based access to the contextual option menu.
The present invention has been described with reference to a preferred embodiment and number of specific alternative embodiments. However, it will be appreciated by those skilled in the relevant fields that a number of other embodiments, differing from those specifically described, will also fall within the spirit and scope of the present invention. Accordingly, it will be understood that the invention is not intended to be limited to the specific embodiments described in the present specification, including documents incorporated by cross-reference as appropriate. The scope of the invention is only limited by the attached claims.
The present application claims the right of priority under 35 U.S.C. §119(e) based on U.S. Provisional Patent Application No. 60/974,077, filed Sep. 21, 2007 which is incorporated by reference herein in its entirety as if fully set forth herein.
| Number | Date | Country | |
|---|---|---|---|
| 60974077 | Sep 2007 | US |
