1. Field of the Invention
This invention relates to systems and methods for interacting with physical documents and a computer, and more specifically to relating user interactions between a physical document and related content on a computer in a hybrid paper and computer-based interface.
2. Description of the Related Art
Paper and computers are the two most commonly used media for document processing. Paper is comfortable to read and annotate, light to carry, flexible to arrange in a space, robust to use in various settings, and well accepted in social settings. Computers are useful in multimedia presentations, document editing, archiving, sharing and search. Because of these unique and complementary advantages, paper and computers are extensively used in parallel in many scenarios. This situation will likely continue in the foreseeable future, due to the technical difficulties and cost-efficiency concerns about completely replacing paper with computers.
In a typical workstation setting, a user may desire simultaneous use of paper and computers, especially by using paper documents 112 and a computer 106 side by side on a table, as shown in
The problem, however, is that the existing technology for mixed use of paper and computers does not provide for convenient transition or interaction between the two media. The content on paper is insulated from computer-based digital tools such as remote sharing, hyperlinks, copy-paste, Internet searching and keyword finding. This gap between paper and computers results in low efficiency and degraded user experience when using paper in combination with a computer. For example, it is tedious for business people to manually transcribe paper receipts for reimbursement, and for accountants to compare the reimbursement form and the original receipts for verification. In another example, it is nearly impossible for a person to search the Internet for an unknown foreign word in a book if he/she does not know how to type in that language. Similarly, it is inconvenient to copy a picture from a paper document to a digital document on a computer.
Efforts have been made to address the paper-computer boundaries, but the work still does not bridge the gap. First, most of the current systems such as PlayAnywhere (Wilson, A. D., PlayAnywhere: a compact interactive tabletop projection-vision system, Proceedings of UIST '05, pp. 83-92), DocuDesk (Everitt, K. M., M. R. Morris, A. J. B. Brush, and A. D. Wilson, DocuDesk: an interactive surface for creating and rehydrating many-to-many linkages among paper and digital documents, Proceedings of IEEE TABLETOP '08, pp. 25-28) and Bonfire (Kane, S. K., D. Avrahami, J. O. Wobbrock, B. Harrison, A. D. Rea, M. Philipose, and A. LaMarca, Bonfire: a nomadic system for hybrid laptop-tabletop interaction, Proceedings of UIST '09, pp. 129-138) focus on interaction with a whole page or document, and do not support fine-grained manipulation within the document (e.g. individual words, symbols and arbitrary regions). Second, those systems only support limited digital functions on paper, typically page-level hyperlinks (PlayAnywhere, DocuDesk), spatial arrangement tracking (Kim, J., S. M. Seitz, and M. Agrawala, Video-based document tracking: unifying your physical and electronic desktops, Proceedings of UIST '04, pp. 99-107), and text transcribing (Newman, W., C. Dance, A. Taylor, S. Taylor, M. Taylor, and T. Aldhous, CamWorks: A Video-based Tool for Efficient Capture from Paper Source Documents, Proceedings of IEEE Multimedia System '99, pp. 647-653; and Wellner, P., Interacting with paper on the DigitalDesk, Communications of the ACM, 1993. 36(7): pp. 87-96), which are not enough to address the above issues. Third, they may interfere with the existing workflow, due to their inflexible hardware configuration and the requirement in some for specially marked paper (Song, H., Guimbretiere, F., Grossman, T., and Fitzmaurice, G., MouseLight: Bimanual Interactions on Digital Paper Using a Pen and a Spatially-aware Mobile Projector, Proceedings of CHI '10).
As described above, current systems for relating paper documents to activities on a computer suffer from numerous limitations, and as such, there is a need for improvements to the ability to work with physical documents and computers at the same time.
Systems and methods described herein provide for interacting with physical documents and at least one computer, and more specifically to providing detailed interactions with fine-grained content of physical documents that is integrated with operations on at least one computer to provide for improved user interactions between the physical documents and the computer.
In one aspect of the invention, a system for interacting with physical documents and at least one computer comprises a camera processing module which processes the content of at least one physical document and detects user interactions on the at least one physical document; a projector processing module which provides visual feedback on the at least one physical document; and a computer with a screen which coordinates the user interactions on the at least one physical document with an action on the computer.
In another aspect of the invention, the camera processing module processes fine-grained content of the at least one physical document, including individual words, characters and graphics, and detects user interactions relating to the fine-grained content.
In another aspect of the invention, the visual feedback provided by the projector processing module is based on user interactions on the physical document.
In another aspect of the invention, the user interactions further include gestures made on the at least one physical document which correspond to actions on the computer.
In another aspect of the invention, the gestures correspond to pre-configured commands which result in a specific type of visual feedback.
In another aspect of the invention, a user interaction on the computer is translated into visual feedback provided by the projector processing module to the at least one physical document.
In another aspect of the invention, the projector processing module provides visual feedback on a physical surface other than the physical document.
In another aspect of the invention, the system further comprises a portable, integrated camera and projector with a foldable frame and at least one mirror, the mirror attached to the frame and positioned over the at least one physical document to reflect an optical path of the camera and projector onto the at least one physical document.
In another aspect of the invention, the camera processing module processes the content of the at least one physical document and obtains a corresponding digital document to display on the computer screen.
In another aspect of the invention, the user interactions on the at least one physical document result in corresponding interactions on the corresponding digital document.
In another aspect of the invention, the camera processing module processes the content of the at least one physical document and obtains digital content which relates to the at least one physical document.
In another aspect of the invention, a method for interacting with at least one physical document and at least one computer comprises processing the at least one physical document; detecting user interactions with the at least one physical document; providing visual feedback on the at least one physical document; and coordinating the user interactions on the at least one physical document with interactions on a computer with a screen.
In another aspect of the invention, the method further comprises processing the at least one physical document to identify fine-grained content, including individual words, characters and graphics; and detecting user interactions relating to the fine-grained content.
In another aspect of the invention, the visual feedback is based on user interactions on the physical document.
In another aspect of the invention, the user interactions further include gestures made on the at least one physical document which correspond to actions on the computer.
In another aspect of the invention, the gestures correspond to pre-configured commands which result in a specific type of visual feedback.
In another aspect of the invention, the method further comprises providing visual feedback on a physical surface other than the physical document.
In another aspect of the invention, the method further comprises translating a user interaction on the computer into visual feedback on the at least one physical document.
In another aspect of the invention, the method further comprises translating user interaction with the at least one physical document with simultaneous user interaction on the computer to manipulate detailed content of the at least one physical document.
In another aspect of the invention, the detailed content of the physical document is manipulated by user interactions using a first hand to interact with the at least one physical document and a second hand to interact with the computer.
In another aspect of the invention, the detailed content of the digital document is manipulated by user interactions using a first hand to interact with the at least one physical document and a second hand to interact with the computer.
In another aspect of the invention, the method further comprises synchronously manipulating detailed content of the physical document and a digital document on the computer using a first hand to interact with the at least one physical document and a second hand to interact with the digital document.
In another aspect of the invention, the method further comprises processing the content of the at least one physical document and obtaining a corresponding digital document to display on the computer screen.
In another aspect of the invention, the user interactions on the at least one physical document result in corresponding interactions on the corresponding digital document.
In another aspect of the invention, the method further comprises processing the content of the at least one physical document and obtaining digital content which relates to the at least one physical document.
In still another aspect of the invention, a computer program product for interacting with at least one physical document and a computer is embodied on a computer readable storage medium, and, when executed by a computer, performs the method comprising processing the at least one physical document; detecting user interactions with the at least one physical document; providing visual feedback on the at least one physical document; and coordinating the user interactions on the at least one physical document with interactions on a computer with a screen.
Additional aspects related to the invention will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. Aspects of the invention may be realized and attained by means of the elements and combinations of various elements and aspects particularly pointed out in the following detailed description and the appended claims.
It is to be understood that both the foregoing and the following descriptions are exemplary and explanatory only and are not intended to limit the claimed invention or application thereof in any manner whatsoever.
The accompanying drawings, which are incorporated in and constitute a part of this specification, exemplify the embodiments of the present invention and, together with the description, serve to explain and illustrate principles of the invention. Specifically:
In the following detailed description, reference will be made to the accompanying drawings. The aforementioned accompanying drawings show by way of illustration and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other implementations may be utilized and that structural changes and/or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense. Additionally, the various embodiments of the invention as described may be implemented in the form of software running on a general purpose computer, in the form of specialized hardware, or combination of software and hardware.
Embodiments of the invention disclosed herein provide for interacting with physical documents and a computer, and more specifically to providing detailed interactions with fine-grained content of physical documents that is integrated with operations on a computer to provide for improved user interactions between the physical documents and the computer. Embodiments of the invention also support two-handed fine-grained interaction with physical documents and digital content using a hybrid camera-projector interface.
One embodiment of the system 100, illustrated in
The camera and projector may be integrated into a single, portable camera-projector unit, as illustrated in
The system provides for fine-grained interaction by allowing users to interact with the details of the physical document, including individual words, characters, symbols, icons and arbitrary regions specified by the users. The system additionally supports numerous computer functions on paper. For instance, the users can apply pen or finger gestures on the a paper document to copy and paste text and graphic content from the paper document to the computer, link a word on the physical document to a web page on the computer, search for specific keywords on the physical document or navigate a street level visual map on the computer by pointing to specific places on a paper map. All of these embodiments are detailed below.
Based on the fine-grained interaction with the physical document, the system supports cross-media two-handed interaction with the physical document and the computer, which combines the complementary affordances of paper and the computer. For instance, if camera-based user interaction with a physical document using a finger or pen is relatively coarse and unreliable, this interaction can be augmented with high fidelity and robust keyboard or mouse input on the computer. In another embodiment, the finger or pen input on the physical document can also be combined with mouse or keyboard input on the computer for multi-pointer operations on the computer. With this hybrid cross-media interaction, the system makes further advances in bridging the paper-computer boundary.
The framework of the system will now be further described, followed by more details of the components of the system. Further details of the interactions enabled by the framework as well as a demonstration of various applications will then be provided.
The system acts as a bridge between the physical document workspace 110 and a digital document workspace 114, as illustrated in
If needed, the paper-computer coordinating processor 116 coordinates actions in the physical document workspace 110 with the digital document workspace 114, manipulating the digital copy 118 or other content on the computer 106. In
A method for interacting with the physical document and the computer is also described and illustrated in the block diagram in
The system described herein provides unique processing of generic document recognition, fine-grained document content location, precise projection correction and two-handed hybrid paper-computer input—all of which will be described in more detail below.
In one embodiment, the camera and projector may be integrated into a combined camera-projector unit 122, as shown in
The camera processing module is responsible for recognizing the physical document, including the content, as well as tracking the movement of the document in order to adjust the visual output of the projector. The camera processing module also performs finger and pen tip detection and tracking as well as performing a coordinate system transform, which is described in more detail below. To be compatible with existing practices, a content-based document recognition algorithm is adopted to recognize paper documents in the camera view. In one embodiment, a color-based algorithm is employed to detect and track a bare finger or a pen tip as distinguished from the physical document. Based on this analysis, the finger or pen interaction with the physical document may be mapped to mouse-pointing operations on the corresponding digital version of the document being displayed on the computer screen. For real-time processing, the slow and accurate recognition algorithm is combined with a fast and relatively inaccurate inter-frame tracking algorithm. The relatively accurate recognition is performed upon user request or automatically at fixed intervals of time (e.g. 1˜2 seconds). Based on the result, the precise location of a paper document in a camera-captured video frame is estimated with the tracking result between two consecutive frames. Every recognition session resets the tracking module to reduce the accumulated error. The tracking algorithm could be based on optical flow or corner features of the camera images. In one embodiment, the algorithm used may be similar to that disclosed in “Video Puppetry: A Performative Interface for Cutout Animation, in ACM Transaction on Graphics, Vol. 27, No. 5, Article 124, 2008, by Barnes et al.,” although one of skill in the art will appreciate that other algorithms may be used for tracking the location and movement of the document.
Embodiments of the system leverage a content-based document image recognition approach, identifying a normal generic printed document as is—without the need for barcodes or special digital paper. In this way, the system is completely compatible with existing document processing practices and provides for wide usability, as any type of document—from a newspaper to a receipt to a standard printout—can be used. Several algorithms may be used for document image recognition, but in this embodiment, we select a Fast Invariant Transform process, known as FIT, as described in Liu, Q., H. Yano, D. Kimber, C. Liao, and L. Wilcox; High Accuracy And Language Independent Document Retrieval With A Fast Invariant Transform; Proceedings of ICME '09, incorporated herein by reference in its entirety. FIT is a generic image feature descriptor, and is thus applicable to a wide range of document types (e.g. text, graphics and photos) and language-independent. FIT is also efficient in terms of search time and feature storage. FIT exploits local features at key points, being robust to partial occlusion, luminance change, scaling, rotations and perspective distortion.
In one embodiment of the system, when a user prints a document, a special instrumented printer drive intercepts the document and sends it to a server, which identifies feature points in every page and calculates a 40-dimension FIT feature vector for each point. The vectors are clustered into a tree for an ANN (Approximate Nearest Neighbor) correspondence search. Other metadata such as text, figures and hot spots in each document page are also extracted and indexed at the server. The same feature calculation is applied to a subsequent query image, and the resulting features are matched against those in the tree. If a feature point from the query image is similar (with some numeric similarity measurement) to a feature point from the index, the two points are matched and they are deemed to be “corresponded.” The page with the most matches (if above a threshold) is taken as the original digital page for the image.
In one embodiment, color-based methods track the tip of a finger or the tip of a pen based on the color of the finger or pen as contrasted with the background, which is typically the physical document itself. The color-based method assumes that the color of the finger and pen tip is distinguishable from the background. For finger tip detection, a fixed color model is adopted for skin color detection; for pen tip detection, a pre-captured pen-tip image for hue histogram back-projection is used. Additional methods may be used as well, as known to one of skill in the art.
To reduce the noise in the position of the detected point, Pt, a post-filter as applied to the Pt values and the Pt is only updated if the tip movement is above a threshold. Moreover, to avoid finger and pen occlusion, the idea of setting the projected cursor at a fixed distance above the detected tip may be used. Since there is a similarity in pen and finger tip processing, the pen-related techniques described below are applicable to finger interaction unless noted otherwise.
In the system described herein, there are many known solutions to realizing touch detection for pen and fingers. Known methods include approximating the finger to surface distance using the finger's shadow, and, as already described, spreading a thin sheet of diffused laser light just above the table for easily detecting objects close to the table.
To interpret, at fine granularity, pen-paper interaction captured by the camera (e.g. pointing with a pen to a word on a paper document), a precise coordinate transform should be established from at least one camera image to at least one identical digital document page. This enables the accommodation of varying printing styles and spatial arrangement of paper sheets. Existing systems detect the boundary of a piece of paper and map the enclosed quadrangle to a rectangular digital image. This method is good enough for coarse granularity interaction, such as projecting a video onto a blank paper sheet. However, it is not accurate enough for word-level and symbol-level interaction, because the margin around the printout may lead to inaccurate mapping between the printed content 112 and the corresponding digital document page 118, as illustrated in
To address the limitations of the existing systems, we exploit the correspondence between the feature points in a camera image and those in the recognized digital document page to derive a homographic transform Hr between a camera reference frame 136 and a recognized digital document reference frame 138, as illustrated in
In one embodiment, to support interaction with arbitrary points on the physical document workspace in general, which may not necessarily be within a paper document, an anchor-pad 144 is utilized to define a table reference frame. The anchor-pad 144 may be a rectangular dark paper sheet of a known size, whose four corners define four points of fixed coordinates (e.g. (1,1), (1,2), (2,1) and (2,2)) in the table reference frame. During calibration, the camera detects the four corners of the anchor pad in its view, and derives a homographic transform Hc between the table, or physical document space 110, and the camera reference frames 136, as illustrated in
Supporting real-time interaction on paper may require an image processing speed of more than 15 frame-per-second (fps). However, the system described herein currently supports approximately 1 fps due to its high computational complexity. In contrast, document tracking techniques such as optical flow can estimate the relative movement of pages in real-time, but with accumulated errors. Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene. See Burton, Andrew and Radford, John; Thinking in Perspective: Critical Essays in the Study of Thought Processes; Routledge; 1978; ISBN 0416858406. The document recognition and document tracking can be combined for hybrid document tracking. In one embodiment, the system periodically recognizes a video frame and derives an Hr. Based on the result, Hr for subsequent video frames is estimated with the optical flow between two consecutive frames. Every recognition session resets the optical flow detection to reduce the accumulated error.
The projector 104 enables dynamic visual feedback directly on the physical document 112 and physical document workspace 110. There are two basic projection types, namely local projection and global projection.
With local projection, the projected image 146 is always aligned with the printout reference frame of a paper document 112 as illustrated in
The local projection usually results from pen-paper interaction, which is first mapped to a pointer operation in the corresponding digital document reference frame. The feedback information for the projector is thus defined directly in the same reference frame. For instance, as shown in
The hardware settings are advantageous in establishing the mapping. The relative positions of the camera, projector and the table surface are fixed and the table is assumed to be flat, so a fixed homographic transform Hp exists between the camera reference frame 136 and the projector reference frame 148. As a result, the document-to-projector mapping can be described as Hp−1*Hr−1. In one embodiment, Hp is derived with a simple one-time calibration, where a pre-stored image with a known pattern is projected to the table surface and captured by the camera. By finding the feature correspondence between the projected and captured images (with N>=4 correspondence pairs), the Hp value is obtained.
The projection transform builds on the content-based camera-document transform. It varies for different document pages (multiple document pages could be recognized in one video frame) and different positions of a moving document in the camera view. The projection transform is also immune to the printing margin, N-up printing and partial occlusion. This immunity of projection transform is critical for precisely aligning the projected visual feedback 146 with the underlying paper document details.
In contrast to local projection, global projection aligns the projection 146 with the table reference frame 110, and is not affected by paper movement. It is usually adopted for some global information that is not associated with a specific document page, such as the creation time of the whole document and the related references. It can be also used as a peripheral display to extend the computer display, for applications such as email notification, an instant message dashboard, or a system performance monitor.
The main issue of global projection is known as keystoning, where the projected image suffers from perspective distortion because of the misalignment of the projector's optical axis and the projection plane's normal, or direction perpendicular to the projection plane. In one embodiment, this can be corrected with reverse-distortion on the projected image 146. The key is to establish the coordinate transform from the projection plane 110 (i.e. the table) to the projector reference frame 148. As described above, the table-to-camera transform Hc and the projector-to-camera transform Hp is already known, so the table-to-projector homographic transform can be derived from Hp−1*Hc.
Based on the underlying camera-projector input/output component, embodiments of the system provide interaction techniques for fine-grained document content manipulation on paper to achieve a computer-equivalent user experience without sacrificing the flexibility and advantages of the paper document. In one embodiment, it is possible to provide cross-media two-handed interaction by mixing the camera input from a first hand in the physical document space with keyboard and mouse input from a second hand to manipulate the digital document space. This two-handed interaction further integrates paper and computers as a closely coupled interactive space.
In one embodiment, the system 100 maps the pen tip input 142 from paper 112 to the corresponding digital document 138 and projects the visual feedback 146 onto paper. With this mechanism, the paper documents and physical document workspace are treated like a touch-sensitive display, so that conventional pen or stylus-type computer operations are extended to the physical document.
In one embodiment, the pen input may be interpreted as either free-form handwriting or command gestures, depending on whether the current input mode is “ink” or “gesture,” respectively. In “ink” mode, the input is recorded as written annotations, which can be stored on a corresponding digital document and retrieved later for review, or shared with remote co-workers viewing the digital document over a network. If an actual inking pen is used, the ink left on the paper usually provides higher fidelity than the digital version, so in an alternate embodiment, ink lifting techniques may be adopted to extract the ink annotations from paper. In “gesture” mode, the pen input is used to construct computer commands, which consist of one or more document segments as target sections for the command and a desired action to be carried out on the document segment. Users may draw pen strokes on the physical document to select individual words, characters, symbols, images, icons and arbitrary regions or shapes for various functions.
Like a normal pen-based interface, there are two basic statuses of the input, namely “hover” and “touch.” According to one embodiment, in “hover” status, the pen is above paper without touching the surface. The user can move the pen to direct a projected cursor to the intended word. At any time, the word closest to the pointer 142 is highlighted 146 by the projector feedback, as shown in
There are many types of gestures for selecting words, symbols and other document content. As illustrated
In one embodiment, the system does not support multi-stroke gestures nor perform gesture recognition in order to provide a simpler implementation. However, the system can support these features if desired. In this embodiment, users need to choose a gesture type manually before issuing a gesture.
To implement the above operations, metadata is extracted from each digital document stored in a system database. Such metadata may include the bounding box (position and size) in the document reference frame of words, characters and icons, and their text and associated uniform resource locations (URLs), if any. The metadata is combined with the pen input to set command targets (e.g. the words selected by an underline gesture), and is also used to generate visual feedback on the paper, such as a rectangular white block to highlight the selected words, as shown in
The projected feedback in response to the gestures is specially designed to limit the possible interference with the original visual features of the paper documents, otherwise the accuracy of the physical-digital interaction mapping could be compromised. First, rendering the gesture strokes is avoided if possible. For example, feedback is only projected for the text selected by the Underline, Bracket, and Vertical gestures, but is not rendered for the raw gesture strokes. Second, thin straight line segments are used for projection (except the lasso and free-from gestures) as much as possible, because they generate fewer feature points than complex patterns. Third, highlighting large areas with solid bright colors is avoided, as the resulting glare may distort the original document's visual features. Lastly, in one embodiment, projected feedback is only placed on the most outer contour 175 of the selected content 177, as illustrated in
In
One solution is adaptive placement of the menu, where the embodied system automatically projects the menu 178 in an area with minimum occlusion. In one embodiment, this is implemented by searching for a region with the least texture and shortest distance from the command target within the projection area. Since it is possible that no regions satisfy both criteria, a weighting function is adopted to choose the optimum region. The spatial distribution of the text could be approximated by that of the previously described FIT feature pointers 182 of the camera images, as illustrated by the dots in
In a situation where there is no good place for the menu, the command action menu may instead be displayed on the computer screen, which is immune to the occlusion issue. The menu may be rendered at a fixed location on the screen for consistent user experience. Moreover, rendering the menu on the screen does not necessarily increase the eye-focus switching between the paper and the screen, as the user usually needs to turn to the computer screen for the results of the command targets executed on the paper document.
The above-described fine-grained interaction relies on accurate document recognition and coordinate transform. Sometimes, however, the recognition may fail due to bad lighting conditions, paper distortion and non-indexed documents. And the transform matrices may be inaccurate due to insufficient feature point correspondence. To recover from such errors, the computer may be exploited to enhance the paper interaction.
If the paper document recognition fails (i.e. the number of matched feature points is below a threshold), one embodiment of the system allows the user to choose the corresponding digital version from a top-N list, or from the whole database. In case of a non-indexed document which is not present in the database, the user switches the camera to a still image mode, takes a high-resolution photograph of the document, and manually indexes it in the database. The system may also apply optical character recognition (OCR) to the picture to generate text metadata.
If the corresponding digital version of the physical is found and the accuracy of the transform matrix is not sufficient (based on an estimate of the number of matched feature points), the system resorts to a digital proxy technique, which uses the paper document for initial coarse interaction and the computer for fine interaction. As shown in
The finger or pen gestures described above can also be applied on the computer as well. In one embodiment of a method for applying gestures on the computer (not illustrated), once the finger or pen gesture operation is done, the user moves the first hand out of the camera view. In response, the digital proxy window shrinks to an icon, and the screen restores to the previous status for the next step of the cross-media operation, for example pasting a copied figure into another document file. Since manipulation of the paper document is bypassed, an inaccurate transform Hr is not as significant.
VII. Two-Handed, Simultaneous Interaction with Physical and Digital Documents
As found in previous studies looking at a worker's manipulation of documents, a worker involved in using documents spends almost half of the time working on multiple documents—for referring, comparing, collating, summarizing and so on. In a situation with a portable computer with a limited screen size, paper documents are often used to extend the screen for multi-document interaction. This interaction, however, is more complicated than normal multi-window operations on a screen, as the documents may reside in different media and involve different input methods. For example, a user may want to copy a figure from paper to the computer, associate a web page with a word on paper, or navigate a street view map on a computer to find a place on a paper map. The input devices for paper are mainly a finger or a pen, and for the computer, a keyboard and a mouse. For these cross-media multiple document operations, one-handed interaction requires the user to switch input devices and sometimes to change body pose, which is inconvenient.
Therefore, one embodiment of the invention supports cross-media two-handed interaction, so that users can use one hand to carry out operations on paper and the other hand to carry out operations on the computer. The two input streams, from the camera and computer, are coordinated to support multiple-document manipulation.
In one embodiment of a method for cross-media interaction, the cross-media two-handed interaction can be used to support information transfer. For instance, to get information on an unfamiliar Japanese word, “” appearing on a paper document, the user may point her first hand to the characters or word, and then use her second hand to choose a command on the computer, such as “search the web.” In response, the system forwards the selected text to the computer, which performs a web search and displays the results to the user. Similarly, the user can easily lasso a picture on the paper document and then copy it into a word processing or other document on the computer. In another embodiment, the information transfer can be in the reverse direction. Multimedia annotations can be projected onto the paper document from the computer. The annotations can be represented by an icon projected on the paper and re-played with a double click. The two hands can also be used to naturally establish information association linking two document segments across the paper-computer boundary. For example, the user can link an encyclopedia or dictionary web page to the Japanese word on the paper, so that selecting the Japanese word on the paper in the future will result in displaying the linked web page on the computer screen. The user can also operate on different views of the same compound document synchronously for multiple-view manipulation. For example, as illustrated in
The two-handed input can be used not only for cross-media operations, but also for single media operations. The system supports augmenting paper operations with the computer input. This is motivated by the complementary affordances of the camera-projector unit and the computer. The camera-based finger input, although natural for paper manipulation, is usually less robust and has a lower input sampling rate than the mouse and keyboard. This causes relatively inferior user experience for paper interaction, especially for fine-grained interaction. The problems with finger or pen input may be magnified when there is only one hand for gesturing on paper (e.g. during two handed cross-media interaction), because, with the other hand providing input to the computer, the friction caused from the finger-paper contact may cause undesired movement of the paper sheets.
To make the best use of the available affordances of the hybrid system, in one embodiment, the keyboard and mouse input may be re-directed provide input and feedback to the paper document, and combine the input with the camera input for two-stage, progressive, fine-grained interaction. For example, as illustrated in
A computer keyboard (not shown) can be also used to add high fidelity text information to paper documents. For example, the user can select a document segment on paper and then type text annotations for the segment; one can also use a keyboard to correct OCR errors for a selected paper document region. This keyboard input is particularly useful, in one example, for a semi-automatic paper receipt transcription application, as described below with respect to
IX. Two-Handed Interaction with Physical and Digital Documents Simultaneously
A fused camera input and computer input can be also applied to screen-only interaction in an additional embodiment. The system can redirect the pen-based or finger-based pointing on the paper document to the computer in order to control digital documents. The pen-based and finger-based pointing can be combined with the mouse input for multi-pointer interaction on the screen without the need for extra hardware. For example, with a physical document-based pointer and a computer-based pointer, a user can scale and rotate a picture simultaneously. In another example, as illustrated in
The interaction techniques described in the variety of embodiments above can be applied to a number of scenarios for mixed use of paper and computers. Several non-limiting examples include paper receipt processing, document manipulation and map navigation, as will be described in more detail immediately below.
Paper receipts are extensively used for their simplicity, robustness and compatibility with existing paper-based work flows. However, integrating paper receipts into new digital financial document work flows is tedious and time-consuming. Much research and various commercial products have been developed in this area. However, many of them require fully manual transcription of information from the receipt, such as expense amounts and dates. Others apply OCR to automatically extract the information from receipts, but lack of a convenient error correction interface and other limitations makes accountants' verification difficult.
In one embodiment of a method of receipt processing, the system described above is capable of processing receipts, as illustrated in
As demonstrated above, the system helps users perform many fine-grained document operations on paper. Keyword finding, copy-paste, and Internet searching are three non-limiting examples. In one embodiment of a keyword finding application, illustrated in
Paper maps provide large, robust, high quality displays, but they lack dynamic information available on a digital map, such as street view images and dynamic traffic information. In one embodiment of the system, illustrated in