Patent Application
20080282138
Today field markup and data collection is typically done on paper, sometimes with supporting check box inspection lists but usually still on paper. Seldom is the support or benefits of digital technology available in practical and usable tools. Field personnel have frequently found their needs frustrated by the fragility or inadequacy of information technology applied to their work. Even normal conditions, like bright sunlight, let alone austere weather or environment conditions generally preclude the use of computational devices. Rugged laptops are heavy and in most cases cannot be comfortably operated in the field. Smaller portable devices (PDAs) provide very limited screen real-state, and may not be rugged enough in many situations.
Commonly field collection is related to spatial information systems which are printed as spatial diagrams such as blueprints, map or other field reference products. However, data collection does not solely include spatial information systems but also may include user notes in a journal, photographs, dictation and/or video of the location. Currently notes taken in the journal are not linked to photographs, which are further not linked to the spatial diagram they correspond to.
A digital paper system includes an input device such as a touch board, a microphone, a keyboard, a digital ink tablet, and a projector, a writing surface and a writing utensil in which the utensil knows its location in real time when it is marking on the surface. The writing surface may take the form of a digital tablet or digital paper, for example digital paper made by the Anoto Group AB and having an ANOTO® pattern. Various types of conventional digital pens include, but are not limited to, the MAXELL® digital pen, the NOKIA® digital pen, the LEAPFROG FLYFUSION® digital pen, the ANOTO® digital pen, the LIVESCRIBE® PULSE smart pen, and the LOGITECH® digital pen. Besides knowledge of placement location, some digital paper systems also maintain records of information like pressure or time as well as various “state” values such as color or width. Further input devices may combined in order to facilitate multi-modal interaction within the computer system. A human operator can issue instructions and provide information via microphone and keyboard. The operator can receive information via the audio speaker. A projector projects images on the map that are visible to the operator.
Methods and systems for multimedia object association to digital paper spatial diagrams are disclosed herein. A method for multimedia object association to digital paper spatial diagrams includes the steps of (1) capturing a media object to be imported into a spatial information system; (2) linking the captured media object to a digital paper document containing a spatial diagram, using a input device configured to select a linking action area located in a first area of the digital paper document and configured to select a location for the captured multimedia data object on a second area of the digital paper document; (3) transferring the media object and the data from the input device to a spatial information system; and (4) combining the media object and the data of the input device to create an updated digital paper document containing the media object.
The preferred and alternative embodiments of the present invention are described in detail below with reference to the following drawings.
In the following description, certain specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, one skilled in the art will understand that the invention may be practiced without these details or with various combinations of these details. In other instances, well-known systems and methods associated with, but not necessarily limited to, spatial information systems, digital paper systems, digital pens and methods for operating the same may not be shown or described in detail to avoid unnecessarily obscuring descriptions of the embodiments of the invention.
The present invention is generally directed to linking data from a multimedia capture device such as a photos, videos, audio, and tags to a digital paper-enabled product for digital data collection using digital paper systems, such as maps, blueprints, diagrams, CAD drawings, and other documents having spatially relevant information printed or applied thereon. The digital paper-enabled product allows a user to create, import, edit, share, and act on paper-based data in and between spatial information systems. In one embodiment the plurality of media objects are associated with particular locations in a spatial information system. For example a note written with a digital pen within a digital paper enabled notebook is linked with a particular location on a digital-enabled product containing a map, sometimes called a spatial diagram.
At least one embodiment of the present invention relates generally to linking a media object with a digital paper-enabled product having digital paper with a digital pattern that spatially cooperates with one or more features or boundaries printed on the pattern, and more specifically to spatial information systems that communicate with a writing instrument previously used to effect changes to the digital paper-enabled product and communicate with a multimedia device used to capture a media object. The spatial information system further combines the media object with the selected location within the digital paper-enabled document. In one embodiment of its product form, the digital paper-enabled product may take the form of a sheet of digital paper bearing a digital pattern and where features (e.g., such as map features) or a spatial reference frame printed onto the digital pattern and where at least one action palette is printed onto the digital pattern, correlated therewith, and usable to effect changes to the features and to link media objects with the digital paper-enabled product. In alternate embodiments, the at least one action palette may be printed onto a separate digital paper-enabled product having a digital pattern. By way of example, a user may take a picture of an area shown on the digital paper-enabled product, then the user may use a writing instrument or indication device, such as a digital pen, to link the picture to the document through executable actions produced by interaction of the writing instrument with the digital paper-enabled product.
In yet another aspect of the invention, a method of processing information obtained from a digital paper-enabled product and capture device includes (1) receiving information with a input device that indicated a link of at least one media object to the digital paper-enabled product, the at least one media object correlated to a location on the digital paper; (2) placing the input device in communication with a spatial information system configured to manage the at least one media object link; (3) transferring the information from the input device to the spatial information system to update the linked media object within the spatial information system; (4) capturing the linked media object from a capture device; and (5) synchronizing the spatial information system to generate an updated version of the digital paper-enabled product.
And still yet in another aspect of the invention, a method of using a digital paper-enabled product when de-coupled from a spatial information system includes the steps of (1) accessing the digital paper-enabled product having a sheet of digital paper bearing a digital pattern and having at least one identifiable region spatially correlated to the digital pattern with a coordinate system; (2) indicating on the product linking action causing a media object to be linked to a spatial information system; and (3) recording the linking action with an indicating device.
At block 120 a media object is associated to a link within the spatial diagram. Association refers to the means by which specific media objects are connected to individual links. One or more media objects may be associated with any link. Conversely, only a subset of the media objects that have been collected may be associated to drawing links. For example pages of notes, some of which, are associated with locations in a spatial diagrams others are not. In one embodiment associations can be established by an explicit selection of a user interface action on the media object, for example by ticking a checkbox on a note page indicating it is to be associated with a link, or selecting an association interface option on a capture device such as a camera or similar device. In an alternate embodiment, associations can be established by using timestamps to associate media objects captured on the temporal vicinity of a linking action, for example by associating with a link those objects whose timestamps fall within plus/minus seconds of the linking action. In yet another alternate embodiment, associations can be established by using identifiers, such as photo or page numbers that are matched to identifiers specified in the linking symbol. In yet another alternate embodiment, associations can be established by location information that may be embedded in the media object, for example GPS location embedded in a photo. The association in this case preferably corresponds to the automatically created link location. In an alternate embodiment linking may occur using a gesture, a voice command, or a multi-modal instruction.
At block 130 the media object is visualized within an electronic representation of a spatial diagram. For example after a user has retrieved the media object from the media device into a computer a user can see the media object on the spatial diagram. In one embodiment linked and/or associated media objects may be presented on an electronic representation of a spatial diagram as icons or thumbnails. A selection action causes the associated media object to be displayed. In an alternate embodiment an electronic representation of a media object, such as the note or photo, a user interface action causes the spatial diagram location to which the media object is associated to be displayed.
At block 140 media object links and associations are corrected. In one embodiment media object links are corrected on an electronic representation of a spatial diagram. The corrections include, but are not limited to deleting a link, inserting a new link, changing the association between links and media objects. This review may take place within an electronic editing environment or in some embodiments can be performed with a digital pen on digital paper printouts of the spatial diagram and associated media object representations such as printout of note pages, photos or representative images of movies.
At block 150 media objects are managed. Media objects associated to links in a spatial diagram may be stored in separate memory areas, in which case the correlation between object and diagram is represented as connection meta-data, or the media objects may be inserted into the same file as the spatial diagram. A media object that is inserted into the same file as the spatial diagram is self contained and can be conveniently transported via email, or other electronic means as a unit. In one embodiment, object management includes retrieving from the capture devices such as a digital pen or a camera only those objects that have been associated to links to one or more spatial diagrams. For example, photos that are not associated to any link remain in the capture device; similarly, only the electronic ink pertaining to note pages that have been associated to links is removed from the digital pen, and others remain. In one embodiment removal is achieved either by selective removal, when the capture device supports this function or by fully removing and then reinserting those objects that have not been associated with links.
By way of an example and generally referring to
While the preferred embodiment of the invention has been illustrated and described, as noted above, many changes can be made without departing from the spirit and scope of the invention. Accordingly, the scope of the invention is not limited by the disclosure of the preferred embodiment. Instead, the invention should be determined entirely by reference to the claims that follow.
This application claims priority to U.S. Provisional Patent Application Nos. 60/941,332 (filed on Jun. 1, 2007) and is a continuation-in-part of U.S. patent application Ser. No. 12/118,656 (filed on May 9, 2008) which claims priority to U.S. Provisional Patent Application Nos. 60/917,034 (filed on May 9, 2007), 60/969,552 (filed on Aug. 31, 2007), 61/028,498 (filed on Feb. 13, 2008), the subject matter of each are incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 60941332 | Jun 2007 | US | |
| 60917034 | May 2007 | US | |
| 60969552 | Aug 2007 | US | |
| 61028498 | Feb 2008 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12118656 | May 2008 | US |
| Child | 12131852 | US |
