Patent Application
20150116284
This invention relates generally to pen-based computing environments, and more particularly to displaying recorded writing with other contextual content collected in a smart pen environment.
A smart pen is an electronic device that digitally captures writing gestures of a user and converts the captured gestures to digital information that can be utilized in a variety of applications. For example, in an optics-based smart pen, the smart pen includes an optical sensor that detects and records coordinates of the pen while writing with respect to a digitally encoded surface (e.g., a dot pattern). The smart pen computing environment can also collect contextual content (such as recorded audio), which can be replayed in the digital domain in conjunction with viewing the captured writing. The smart pen can therefore provide an enriched note taking experience for users by providing both the convenience of operating in the paper domain and the functionality and flexibility associated with digital environments. However, it is challenging to structure and organize the vast amount of information collected in a smart pen environment to ensure a productive reviewing experience.
The smart pen computing environment may collect large amounts of data in a recording session. The data collected may not naturally fit into a fixed display area, so a portion of the data may be displayed at a time. Zooming functionalities in user interfaces may allow a user to control the amount of data presented, but normal zoom functionalities scale all content at the same rate. Zooming out (or scaling down content items) renders the smallest features on the display illegible while zooming in (or scaling up content items) causes the largest features to dominate the display.
A system and a method are disclosed for smart zooming of content displayed in a pen-based computing system. In one embodiment, a paper strip layout is obtained. The paper strip layout is a digitally displayed representation of a plurality of paper strips, including a first paper strip having a first content and a second paper strip having a second content. A request is received to adjust the zoom level of the first content of the first paper strip. A first transformation is applied to the first content to generate transformed first content based on the the input zoom level. If applying the first transformation to the second content will cause the second content to exceed a boundary of the second paper strip, then a second transformation is applied to the second content of the second paper strip such that the second content fits within the boundary.
In an embodiment, the first transformation includes a first scaling component and a second shifting component. If applying the first transformation to the second content causes the second content to exceed a boundary of the second paper strip, then a second shifting component different from the first shifting component is determined and applied to the second content along with the first scaling component. If applying a second shifting component cannot keep the second content inside the boundary while applying the first scaling component, then a second shifting component and second scaling component are determined and applied to the second content. In an embodiment, the method is performed by a processor that executes instructions stored to a non-transitory computer readable medium.
The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.
In one embodiment, the writing surface 105 comprises a sheet of paper (or any other suitable material that can be written upon) and is encoded with a pattern (e.g., a dot pattern) that can be sensed by the smart pen 110. The pattern is sufficiently unique to enable the smart pen 110 to determine its relative positioning (e.g., relative or absolute) with respect to the writing surface 105. In another embodiment, the writing surface 105 comprises electronic paper, or e-paper, or may comprise a display screen of an electronic device (e.g., a tablet, a projector), which may be the computing device 115 or a different device. In other embodiments, the relative positioning of the smart pen 110 with respect to the writing surface 105 is determined without use of a dot pattern. For example, in an embodiment, where the writing surface 105 comprises an electronic surface, the sensing may be performed entirely by the writing surface 105 instead of by the smart pen 110, or in conjunction with the smart pen 110. Movement of the smart pen 110 may be sensed, for example, via optical sensing of the smart pen 110, via motion sensing of the smart pen 110, via touch sensing of the writing surface 105, via a fiducial marking, or other suitable means.
The smart pen 110 is an electronic device that digitally captures interactions with the writing surface 105 (e.g., writing gestures and/or control inputs). The smart pen 110 is communicatively coupled to the computing device 115 either directly or via the network 120. The captured writing gestures and/or control inputs may be transferred from the smart pen 110 to the computing device 115 (e.g., either in real time or at a later time) for use with one or more applications executing on the computing device 115. Furthermore, digital data and/or control inputs may be communicated from the computing device 115 to the smart pen 110 (either in real time or as an offline process) for use with an application executing on the smart pen 110. Commands may similarly be communicated from the smart pen 110 to the computing device 115 for use with an application executing on the computing device 115. The cloud server 125 provides remote storage and/or application services that can be utilized by the smart pen 110 and/or the computing device 115. The pen-based computing system 100 thus enables a wide variety of applications that combine user interactions in both paper and digital domains.
In one embodiment, the smart pen 110 comprises a writing instrument (e.g., an ink-based ball point pen, a stylus device without ink, a stylus device that leaves “digital ink” on a display, a felt marker, a pencil, or other writing apparatus) with embedded computing components and various input/output functionalities. A user may write with the smart pen 110 on the writing surface 105 as the user would with a conventional pen. During the operation, the smart pen 110 digitally captures the writing gestures made on the writing surface 105 and stores electronic representations of the writing gestures. The captured writing gestures have both spatial components and a time component. In one embodiment, the smart pen 110 captures position samples (i.e., coordinate information) of the smart pen 110 with respect to the writing surface 105 at various sample times and stores the captured position information together with the timing information of each sample. The captured writing gestures may furthermore include identifying information associated with the particular writing surface 105 such as, for example, identifying information of a particular page in a particular notebook so as to distinguish between data captured with different writing surfaces 105. In another embodiment, the smart pen 110 also captures other attributes of the writing gestures chosen by the user. For example, ink color may be selected by tapping a printed icon on the writing surface 105, selecting an icon on a computer display, etc. This ink information (color, line width, line style, etc.) may also be encoded in the captured data.
In an embodiment, the computing device 115 additionally captures contextual data while the smart pen 110 captures written gestures. In an alternative embodiment, written gestures may instead be captured by the computing device 115 or writing surface 105 (if different from the computing device 115) instead of, or in addition to, being captured by the smart pen 110. The contextual data may include audio and/or video from an audio/visual source (e.g., the surrounding room). Contextual data may also include, for example, user interactions with the computing device 115 (e.g. documents, web pages, emails, and other concurrently viewed content), information gathered by the computing device 115 (e.g., geospatial location), and synchronization information (e.g., cue points) associated with time-based content (e.g., audio or video) being viewed or recorded on the computing device 115. The computing device 115 stores the contextual data synchronized in time with the captured writing gestures (i.e., the relative timing information between the captured written gestures and contextual data is preserved). In an alternate embodiment, the smart pen 110 or a combination of a smart pen 110 and a computing device 115 captures contextual data. Furthermore, in an alternate embodiment, some or all of the contextual data can be stored on the smart pen 110 instead of, or in addition to, being stored on the computing device 115.
Synchronization between the smart pen 110 and the computing device 115 (or between multiple smart pens 110 and/or computing devices 115) may be assured in a variety of different ways when capturing contextual information. For example, a universal clock may be used for synchronization between different devices. In an alternate embodiment, local device-to-device synchronization is performed between two or more devices. In another embodiment, content captured by the smart 110 or computing device 115 can be combined with previously captured data and synchronized in post-processing. Synchronization of the captured writing gestures, audio data, and/or digital data may be performed by the smart pen 110, the computing device 115, a remote server (e.g., the cloud server 125) or by a combination of devices.
In one embodiment, the smart pen 110 is capable of outputting visual and/or audio information. The smart pen 110 may furthermore execute one or more software applications that control various outputs and operations of the smart pen 110 in response to different inputs.
In one embodiment, the smart pen 110 can furthermore detect text or other pre-existing content on the writing surface 105. The pre-existing content may include content previously created by the smart pen 110 itself or pre-printed content from other sources (e.g., a printed set of lecture slides). In one embodiment, the smart pen 110 directly recognizes the pre-existing content itself (e.g., by performing text recognition). In another embodiment, the smart pen recognizes positional information of the smart pen 110 and determines what pre-content is being interacted by correlating the captured positional information with known positional information of the pre-existing content. For example, the smart pen 110 can tap on a particular word or image on the writing surface 105, and the smart pen 110 could then take some action in response to recognizing the pre-existing content such as creating contextual data or transmitting a command to the computing device 115. Tapping pre-existing content symbols can create contextual markers associated with recently captured written gestures. Examples of contextual markers can include, for example, indications that the recently captured written gesture is an important item, a task, or should be associated with a particular pre-existing or user-defined tag. As another example, tapping pre-printed content symbolizing controls for a recording device could indicate to the computing device 115 that an associated active audio or video recorder should begin or stop recording. In another example, the smart pen 110 could translate a word on the page by either displaying the translation on a display screen or playing an audio recording of it (e.g., translating a Chinese character to an English word).
The computing device 115 may comprise, for example, a tablet computing device, a mobile phone, a laptop or desktop computer, or other electronic device (e.g., another smart pen 110). The computing device 115 may execute one or more applications that can be used in conjunction with the smart pen 110. For example, written gestures and contextual data captured by the smart pen 110 may be transferred to the computing system 115 for storage, playback, editing, and/or further processing. Additionally, data and or control signals available on the computing device 115 may be transferred to the smart pen 110. Furthermore, applications executing concurrently on the smart pen 110 and the computing device 115 may enable a variety of different real-time interactions between the smart pen 110 and the computing device 115. For example, interactions between the smart pen 110 and the writing surface 105 may be used to provide input to an application executing on the computing device 115 (or vice versa). Additionally, the captured stroke data may be displayed in real-time in the computing device 115 as it is being captured by the smart pen 110.
In order to enable communication between the smart pen 110 and the computing device 115, the smart pen 110 and the computing device 115 may establish a “pairing” with each other. The pairing allows the devices to recognize each other and to authorize data transfer between the two devices. Once paired, data and/or control signals may be transmitted between the smart pen 110 and the computing device 115 through wired or wireless means. In one embodiment, both the smart pen 110 and the computing device 115 carry a TCP/IP network stack linked to their respective network adapters. The devices 110, 115 thus support communication using direct (TCP) and broadcast (UDP) sockets with applications executing on each of the smart pen 110 and the computing device 115 able to use these sockets to communicate.
The network 120 enables communication between the smart pen 110, the computing device 115, and the cloud server 125. The network 120 enables the smart pen 110 to, for example, transfer captured contextual data between the smart pen 110, the computing device 115, and/or the cloud server 125, communicate control signals between the smart pen 110, the computing device 115, and/or cloud server 125, and/or communicate various other data signals between the smart pen 110, the computing device 115, and/or cloud server 125 to enable various applications. The network 120 may include wireless communication protocols such as, for example, Bluetooth, WiFi, WiMax, cellular networks, infrared communication, acoustic communication, or custom protocols, and/or may include wired communication protocols such as USB or Ethernet. Alternatively, or in addition, the smart pen 110 and computing device 115 may communicate directly via a wired or wireless connection without requiring the network 120.
The cloud server 125 comprises a remote computing system coupled to the smart pen 110 and/or the computing device 115 via the network 120. For example, in one embodiment, the cloud server 125 provides remote storage for data captured by the smart pen 110 and/or the computing device 115. Furthermore, data stored on the cloud server 125 can be accessed and used by the smart pen 110 and/or the computing device 115 in the context of various applications.
The marker 205 comprises any suitable marking mechanism, including any ink-based or graphite-based marking devices or any other devices that can be used for writing. The marker 205 is coupled to a pen down sensor 213, such as a pressure sensitive element. In an alternate embodiment, the marker 205 may make electronic marks on a writing surface 105 using a paired projector or electronic display.
The imaging system 210 comprises optics and sensors for imaging an area of a surface near the marker 205. The imaging system 210 may be used to capture handwriting and gestures made with the smart pen 110. For example, the imaging system 210 may include an infrared light source that illuminates a writing surface 105 in the general vicinity of the marker 205, where the writing surface 105 includes an encoded pattern. By processing the image of the encoded pattern, the smart pen 110 can determine where the marker 205 is in relation to the writing surface 105. An imaging array of the imaging system 210 then images the surface near the marker 205 and captures a portion of a coded pattern in its field of view.
In other embodiments of the smart pen 110, an appropriate alternative mechanism for capturing writing gestures may be used. For example, in one embodiment, position on the page is determined by using pre-printed marks, such as words or portions of a photo or other image. By correlating the detected marks to a digital version of the document, position of the smart pen 110 can be determined. For example, in one embodiment, the smart pen's position with respect to a printed newspaper can be determined by comparing the images captured by the imaging system 210 of the smart pen 110 with a cloud-based digital version of the newspaper. In this embodiment, the encoded pattern on the writing surface 105 may not be needed because other content on the page can be used as reference points. Data captured by the imaging system 210 is subsequently processed using one or more content recognition algorithms such as character recognition. In another embodiment, the imaging system 210 can be used to scan and capture written content that already exists on the writing surface 105. This imaging system can be used, for example, to recognize handwritten or printed text, images, or controls on the writing surface 105. In other alternative embodiments, the imaging system 210 may be omitted from the smart pen 110, for example, in embodiments where gestures are captured by a writing surface 105 integrated with an electronic device (e.g., a tablet) rather than by the smart pen 110.
The pen down sensor 213 determines when the smart pen is down. As used herein, the phrase “pen is down” indicates that the marker 205 is pressed against or engaged with a writing surface 105. In an embodiment, the pen down sensor 213 produces an output when the pen is down, thereby detecting when the smart pen 110 is being used to write on a surface or is being used to interact with controls or buttons (e.g., tapping) on the writing surface 105. Embodiments of the pen down sensor 213 may include capacitive sensors, piezoresistive sensors, mechanical diaphragms, and electromagnetic diaphragms. The imaging system 210 may further be used in combination with the pen down sensor 213 to determine when the marker 205 is touching the writing surface 105. For example, the imaging system 210 could be used to determine if the marker 205 is within a particular range of a writing surface 105 using image processing (e.g. based on a fast Fourier transform of a capture image). In an alternate embodiment, a separate range-finding optical, laser, or acoustic device could be used with the pen down sensor 213. In an alternative embodiment, the smart pen 110 can detect vibrations indicating when the pen is writing or interacting with controls on the writing surface 105. In an alternative embodiment, a pen up sensor may be used to determine when the smart pen 110 is up. As used herein, the phrase “pen is up,” indicates that the marker 205 is neither pressed against nor engaged with a writing surface 105. In some embodiments, the pen down sensor 213 may additionally be coupled with the stylus tip 217, or there may be an additional pen down sensor coupled with or incorporated in the stylus tip 217.
The power status mechanism 215 can toggle the power status of the smart pen 110. The power status mechanism may also sense and output the power status of the smart pen 110. The power status mechanism may be embodied as a rotatable switch integrated with the pen body, a mechanical button, a dial, a touch screen input, a capacitive button, an optical sensor, a temperature sensor, or a vibration sensor. When the power status mechanism 215 is toggled on, the pen's battery 235 is activated, as are the imaging system 210, the input/output device 220, the processor 225, and onboard memory 230. In some embodiments, the power status mechanism 215 toggles status lights, displays, microphones, speakers, and other components of the smart pen 110. In some embodiments, the power status mechanism 215 may be mechanically, electrically, or magnetically coupled to the marker 205 such that the marker 205 extends when the power status mechanism 215 is toggled on and retracts when the power status mechanism 215 is toggled off. In some embodiments, the power status mechanism 215 is coupled to the marker 205 and/or the capacitive tip such that use of the marker and/or capacitive tip 217 toggles the power status. In some embodiments, the power status mechanism 215 may have multiple positions, each position toggling a particular subset of the components in the smart pen 110.
The stylus tip 217 is used to write on or otherwise interact with devices or objects without leaving a physical ink mark. Examples of devices for use with the stylus tip might include tablets, phones, personal digital assistants, interactive whiteboards, or other devices capable of touch-sensitive input. The stylus tip may make use of capacitance or pressure sensing. In some embodiments, the stylus tip may be used in place of or in combination with the marker 205.
The input/output (I/O) device 220 allows communication between the smart pen 110 and the network 120 and/or the computing device 115. The I/O device 220 may include a wired and/or a wireless communication interface such as, for example, a Bluetooth, Wi-Fi, WiMax, 3G, 4G, infrared, or ultrasonic interface, as well as any supporting antennas and electronics.
A processor 225, onboard memory 230 (i.e., a non-transitory computer-readable storage medium), and battery 235 (or any other suitable power source) enable computing functionalities to be performed on the smart pen 110. The processor 225 is coupled to the input and output devices (e.g., imaging system 210, pen down sensor 213, power status mechanism 215, stylus tip 217, and input/output device 220) as well as onboard memory 230 and battery 235, thereby enabling applications running on the smart pen 110 to use those components. As a result, executable applications can be stored to a non-transitory computer-readable storage medium of the onboard memory 230 and executed by the processor 225 to carry out the various functions attributed to the smart pen 110 that are described herein. The memory 230 may furthermore store the recorded written and contextual data, either indefinitely or until offloaded from the smart pen 110 to a computing system 115 or cloud server 125.
In an embodiment, the processor 225 and onboard memory 230 include one or more executable applications supporting and enabling a menu structure and navigation through a file system or application menu, allowing launch of an application or of a functionality of an application. For example, navigation between menu items comprises an interaction between the user and the smart pen 110 involving spoken and/or written commands and/or gestures by the user and audio and/or visual feedback from the smart pen computing system. In an embodiment, pen commands can be activated using a “launch line.” For example, on dot paper, the user draws a horizontal line from right to left and then back over the first segment, at which time the pen prompts the user for a command. The user then prints (e.g., using block characters) above the line the desired command or menu to be accessed (e.g., Wi-Fi Settings, Playback Recording, etc.). Using integrated character recognition (ICR), the pen can convert the written gestures into text for command or data input. In alternative embodiments, a different type of gesture can be recognized to enable the launch line. Hence, the smart pen 110 may receive input to navigate the menu structure from a variety of modalities.
During a smart pen computing session, the pen-based computing system 100 acquires content that comes in two primary forms, that generated or collected through the operation of the smart pen 110, and that generated in or collected by a computing device 115. This data may include, for example, stroke data, audio data, digital content data, and other contextual data.
Stroke data represents, for example, a sequence of temporally indexed digital samples encoding coordinate information (e.g., “X” and “Y” coordinates) of the smart pen's position with respect to a particular writing surface 105 captured at various sample times. Generally, an individual stroke begins when the pen is down and ends when the pen is up. Additionally, in one embodiment, the stroke data can include other information such as, for example, pen angle, pen rotation, pen velocity, pen acceleration, or other positional, angular, or motion characteristics of the smart pen 110. The writing surface 105 may change over time (e.g., when the user changes pages of a notebook or switches notebooks) and therefore identifying information for the writing surface may also be captured in the stroke data.
Audio data includes, for example, a sequence of temporally indexed digital audio samples captured at various sample times. Generally, an individual audio clip begins when a “record” command is captured and ends when a “stop record” command is captured. In some embodiments, audio data may include multiple audio signals (e.g., stereo audio data).
The captured digital content represents states associated with one or more applications executing on the computing device 115 captured during a smart pen computing session. The state information could represent, for example, a digital document or web page being displayed by the computing device 115 at a given time, a particular portion of a digital document or web page being displayed by the computing device at a given time, inputs received by the computing device at a given time, etc. The state of the computing device 115 may change over time based on user interactions with the computing device 115 and/or in response to commands or inputs from the stroke data (e.g., gesture commands) or audio data (e.g., voice commands).
Other data captured by the smart pen system may include contextual tags which stores identifiers associated with content that has been marked in a particular way. For example, a user can tap a button to categorize content according to various content categories (e.g., tasks for follow up, important content, etc.). Photographs or video captured during a smart pen computing session may also be stored and temporally indexed. Geospatial information pertaining to a location where the smart pen computing session took place (e.g., captured using a global positioning system) can also be captured and stored. Furthermore, pairing data or commands executed within the smart pen computing system 100 can be captured and stored.
In one embodiment, a smart computing session starts when a “record” command is captured and ends when a “stop record” command is captured. Alternatively, the smart pen computing session may start automatically when a smart pen computing application is initiated on the computing device 115, or may start and end automatically when the smart pen 110 is turned on and off.
The event timestamp field 310 indicates when in time a particular event occurred. Event timestamps may be with respect to a universal time such as UTC (Coordinated Universal Time), Unix time, other time systems, or any offset thereof, or may be a relative time specified relative to other events or some reference time (e.g., relative to a power on time of the smart pen 110 or computing device 115). Timestamps may be implemented to arbitrary precision. In various possible implementations, timestamps may be stored to indicate the start time of the event, the end time of the event, or both.
The event content field 315 indicates data (or a reference to data) captured by the pen-based computing system 100 such as, for example, written content, recorded audio or video, photographs, geospatial information, pairing data between a smart pen 110 and a computing device 115, digital data clips referencing content concurrently displayed on a computing device 115 during a smart pen computing session, commands to the smart pen 110 and computing device 115, contextual markers, retrieved text and media, web pages, other information accessed from a cloud server 125, and other contextual data.
For example, each stroke captured by the smart pen 110 is generally stored as a separate event and referenced by the event content field 315. Similarly, audio capture events are stored as separate events with the audio clip referenced by the event capture field 315. Changes to the state of an application executing on the computing device during a smart pen computing session may also be captured as an event and referenced by the event capture field 315 to indicate, for example, that the user viewed a particular digital document or browsed a particular web site at a given time during the smart pen computing session. Contextual markers may be stored in the event capture field 315 to indicate that the user applied a particular tag to content. For an event associated with a photograph, the event content field 315 may contain the photographic data or a reference to the file location where the photograph is stored. For an event associated with an audio and/or video file, the event content field 315 may contain the audio and/or video file or a reference to the file location where the audio and/or video is stored.
The metadata field 325 includes additional data associated with the event. Data stored in the metadata field 325 can include, for example, information identifying the source device associated with the event content field 315 as well as relevant state data about that device. For written content consisting of strokes, the metadata field 325 includes, for example, page address information (e.g., surface type, page number, notebook ID, digital file reference, and so forth) associated with the writing surface 105. Metadata associated with a photograph includes, for example, source camera data, the camera application, and applied photo processing. Similarly, the metadata field 325 for recorded audio and video includes, for example, microphone and/or camera data, the recording application, commands input to the recording application, and applied audio and/or video processing. Geospatial information (e.g., Global Positioning System coordinates) can also be included in the metadata field 325 to provide additional contextual data pertaining to the location where the smart pen 110 or computing device 115 was used to capture the event. Metadata associated with events related to concurrently displayed content (such as text, email, documents, images, audio, video, web pages, applications, or a combination thereof) includes, for example, content source and user commands while viewing the concurrently displayed content. Metadata associated with commands and contextual markers includes, for example, information about the writing surface 105 such as surface type, page number, notebook ID, and digital file reference.
Events may contain references to organizational markers referred to herein as “clusters” and “snippets.” A cluster comprises a set of one or more strokes grouped together based on contextual data such as the relative timing of the strokes, the relative physical positioning of the strokes, the result of handwriting recognition applied to the strokes, etc. Generally, each stroke is associated with one and only one cluster. In one embodiment, strokes are grouped into clusters according to a process that is generally intended to generate a one-to-one correspondence between a cluster and a single written word. In practice, the grouping may not always necessarily be one-to-one, and the system can still achieve the functionality described herein without perfect grouping of strokes to words. A process for grouping strokes into clusters is described in further detail below with reference to
A snippet comprises a set of one or more events and may include both strokes and other types of events such as contextual markers, audio, pictures, video, commands, etc. Generally, strokes that are grouped into a single cluster are grouped in the same snippet, but the snippet may also include other clusters. Events are generally grouped together into snippets based on contextual data such as the relative timing of the events, the relative physical positioning of the strokes, the result of handwriting recognition applied to the strokes, etc. In one embodiment, events are grouped into snippets (according to a process referred to herein as “snippetting”) such that each snippet generally corresponds to a complete thought such as a sentence, list item, numbered item, or sketched drawing captured by the smart pen 110 while engaged with a writing surface 105. Generally, events correlated into a snippet have strong temporal correlation, but a later event can be correlated into an earlier snippet if there are strong non-temporal correlations such as, for example, when there is a strong correlation based on spatial location. Furthermore, the automated process for grouping events into snippets need not necessarily be perfect to achieve the functionality described herein. A process for grouping events into snippets is described in further detail below with reference to
In one embodiment, written data associated with clusters and/or snippets may be automatically processed and converted to text using handwriting recognition or optical character recognition. The recognized text may be stored in place of, or in addition to, the stroke data itself in a cluster or snippet.
Although
A particular use case resulting in the example events shown in
The device synchronizer 405 synchronizes data received from various components of the pen-based computing system 100. For example, written data, commands, and contextual markers from the smart pen 110 are synchronized with recorded audio, recorded video, photographs, concurrently viewed web pages, digital documents, or other content, and commands to the computing device 115. Additional contextual data may be accessed from the cloud server 125. The device synchronizer 405 may process data continuously as it is collected or in discrete batches. When the smart pen 110 and computing device 115 are not paired while data is collected, the device synchronizer 405 engine can merge relevant contextual data with written data from the smart pen 110 when the devices are again paired. The device synchronizer 405 processes received data into events, which are stored in the event store 410. In one embodiment, the timestamp 310 is used to organize events in the event store 410 so that events can later be played back in the same order that they are captured.
The event store 410 stores events gathered by the device synchronizer 405. In one embodiment, events comprise various fields such as timestamp 310, event content 315, event metadata 325, an associated cluster 335, and an associated snippet 345, as described above. In one embodiment, the event store 410 indexes events by timestamp. Alternate embodiments may index data by cluster or snippet as a substitute or supplement to indexing by timestamp. The event store 410 is a source of input data for the cluster engine 415 and snippet engine 425.
The cluster engine 415 takes events containing stroke data from the event store 410 and correlates them into clusters. The correlated clusters correspond to aggregated strokes having a particular temporal and/or spatial relationship. For example, a cluster algorithm may cluster strokes such that each cluster generally corresponds to a discrete word written by a user of the smart pen 110, although this is not necessarily the case. In some cases, temporal proximity of strokes is not necessarily required to cluster the strokes. For example, strokes may be clustered based on strong spatial correlation alone. The cluster engine 415 may also apply integrated character recognition (ICR), optical character recognition (OCR), or handwriting recognition to captured strokes and results of these processes may be used in clustering. For example, strokes may be clustered when the cluster engine 415 recognizes a complete word that includes those strokes. The resulting clustered data may be output as indexed strokes, an image representing the aggregated strokes, a digital character conversion of the strokes, or a combination thereof. The output from the cluster engine 415 is stored in the cluster store 420.
The cluster store 420 receives output clusters from the cluster engine 415. In one embodiment, the clusters may be indexed by associated timestamp 310. In other embodiments, clusters may be indexed by associated snippet as a substitute or supplement to indexing by timestamp field 310. The information contained in the cluster store 420 is a source of input data for the snippet engine 425.
The snippet engine 430 takes events from the event store 410 and clusters from the cluster store 420 as inputs. The clusters from the cluster store are correlated according to positional and/or temporal information associated with each cluster. For example, if a user writes horizontally across a writing surface 105, the snippet engine may group clusters arranged across the horizontal row into a single snippet. If a user writes vertically, the snippet engine 430 may group the clusters arranged across the vertical column into a single snippet. If a user sketches a drawing, the snippet engine may group all the strokes of that drawing into a snippet. The snippet engine 425 may group events other than clusters of strokes into snippets. For example, events associated with relevant contextual data may be grouped into a snippet together with related stroke events or clusters to organize the events in a way that captures the thought process of the user while taking notes. For example, if a photograph was taken or a recording started in the middle of or after a snippet, that photograph or recording would be linked to that snippet. In some embodiments, if an audio or video file is being recorded or played during a snippet, that audio or video file is linked to the snippet along with a time position in the file corresponding to the time of the snippet. The times associated with a snippet include the first timestamp field 310, the last contained timestamp field 310, the average of the first and last contained timestamp fields 310, or the average of all contained timestamp fields 310. The output of the snippet engine 425 can includes references to all contained events, strokes, and clusters. In some embodiments, the output of the snippet engine 425 may include a character representation of all contained clusters or an image of all clusters and other content (photographs, preview frames of videos or web pages) in a snippet.
The paper strip display module 435 comprises instructions for displaying snippet information to a user. In one embodiment, all events associated with a snippet are displayed together. In one embodiment, successive snippets are displayed in a temporal order. In one embodiment, the paper strip display module 435 merges snippets collected by, and stored on, multiple devices in the pen-based computing system 100. In alternative embodiments, snippets may be displayed in an order based on the position (on the writing surface 105) of the strokes in the snippet, based on the geospatial location where the snippets were collected, or based on the smart pen 110 that collected the snippet. In an embodiment, the content in a paper strip may be resized using a zoom functionality.
The architecture described herein need not be implemented entirely on the same device. In some embodiments, data may be manipulated or stored across multiple devices in the pen-based computing system 100. Some elements to manipulate or store data may be implemented or duplicated on multiple devices. In an alternate embodiment, the smart pen performs the device synchronization 405, contains the event store 410 and cluster store 420, and also implements the cluster engine 415. Event and cluster information is transmitted over the network 120 to a computing device 115, which implements the snippet engine 425 and contains the snippet store 430. In an alternate embodiment, all information from event stores 410 on the smart pen 110 and computing device 115 are duplicated in a separate event store 410 on a cloud server 125. One skilled in the art can envisage multiple variations on the architecture in
Organizing Stroke Data into Clusters and Snippets
After clusters are correlated 610, snippets are linked 615 to contextual data such as contextual markers, commands, photographs, location information, audio/video recordings, and concurrently viewed web pages, email, and documents. For example, in one embodiment, non-stroke events are retrieved from the event store 410 and linked to snippets according to temporal proximity, spatial proximity, and/or user interactions. For example, a user may indicate that an image is associated with text and therefore should be included as part of the same snippet. In some embodiments, metadata about contextual content such as title, description, or associated tags may be correlated with words in a snippet to associate the contextual content with a snippet. Next, the associated clusters and events in a snippet are stored 620 in the snippet store 430. The snippet engine 425 may then display 625 snippets on a display of a computing device (e.g., computing device 115). If a user disagrees with any of the automated snippet groupings, the user can manually break apart snippets or merge snippets. The snippet engine then receives 630 corrections from the user. These corrected snippets are stored 620 in the snippet store 430.
In cases where a user writes on the writing surface 105 from the beginning of the page to the end of the page, positional and temporal data correlate and thus clustering based on just one of either temporal or spatial proximity may be sufficient. However, when a user skips around the writing surface 105 to make corrections and amplifications to previously written text, positional and temporal data may not correlate. In an embodiment, a stroke received at a later time than proximate strokes may be clustered with proximate strokes if the later stroke spatially intersects or is within a predefined distance of at least one of the proximate strokes. In an embodiment, the later stroke may be grouped in the same snippet as earlier strokes as long as the earlier and later strokes are clustered together. When a later stroke does not spatially intersect earlier proximate strokes, the later stroke may be correlated into a separate cluster from the earlier strokes. Strokes that are correlated into separate clusters from other nearby strokes may be grouped into a separate snippet than the nearby strokes based on lack of temporal correlation. A user may write on a page of the writing surface 105 in two or more distinct recording sessions. In an embodiment, any strokes on the same page of the writing surface 105 are considered for clustering and snippetting regardless of recording session. In an alternate embodiment, the user may specify that writing on the same page be processed for clusters and snippets separately based on position or recording session.
Events captured during a smart pen computing session can be replayed in synchronization. For example, captured stroke data may be replayed, for example, as a “movie” of the captured strokes on a display of the computing device 115. Concurrently captured audio or other captured events may be replayed in synchronization based on the relative timestamps between the data. For example, captured audio can be replayed in synchronization with the stroke data to show what the user was hearing when writing different strokes. Furthermore, captured digital content may be replayed as a “movie” to show transitions between states of the computing device 115 that occurred while the user was writing. For example, the computing device 115 can show what web page, document, or portion of a document the user was looking at when writing different strokes.
In another embodiment, the user can then interact with the recorded data in a variety of different ways. For example, in one embodiment, the user can interact with (e.g., tap) a particular location on the writing surface 105 corresponding to previously captured strokes. The time stamp associated with that stroke event can then be determined and a replay session can begin from that time location.
By grouping captured events into snippets of related content, the user is given even more flexibility in reviewing the data captured during a smart pen computing sessions. For example, in one embodiment, each snippet may be displayed according to its recognized text and organized into lines called paper strips on a display screen. The user can sort paper strips containing snippets based on snippet timestamp so that the snippets appear sequentially even if the corresponding stroke data is organized completely differently on the page. Alternatively, the paper strips containing snippets can be organized based on tags or other user-defined search criteria. If a command or contextual marker is associated with a snippet, then an icon corresponding to that command or contextual marker may be displayed in the same paper strip as the text in that snippet. Selecting an icon corresponding to a command or contextual marker may prompt the user for additional information. For example, selecting an icon associated with a task contextual marker may prompt the user to create a task item from the associated snippet for use within the reviewing application and/or an external application. As another example, selecting an icon associated with a tag contextual marker may prompt the user to input text describing and/or categorizing the associated snippet.
If a photograph is associated with a snippet of written data, a small thumbnail version of the photograph may be displayed in the same paper strip as the rest of the snippet. If a photograph is associated with no other snippet, a version of the photograph larger than a thumbnail may be displayed in a separate paper strip. If a geospatial location or calendar event is associated with a snippet, an icon corresponding to a location or calendar event may be displayed in the same paper strip as the associated snippet, and selection of this icon may link the user to a display of the location on a map or the corresponding calendar entry.
If an audio and/or video recording is associated with a snippet, then selecting a snippet may replay an excerpt of the audio and/or video that is temporally correlated with the written data in that snippet. In one embodiment, continuous playback may be enabled so that selection of a snippet may initiate playback that begins at a time corresponding to the beginning of a snippet. The continuous playback may continue until the end of the recording. In an embodiment, a visual signal may indicate which snippet is temporally correlated with the current position of the audio/video playback. If a webpage, email, or document is associated with a snippet, selecting the snippet may access the associated webpage, email, or document.
In an embodiment, the user can replay notes based on viewing other digital content. For example, suppose a user watches a digital movie on the computing device 115 while taking notes on the writing surface 105. Later, the user can replay the digital movie and see the user's notes replayed while watching a movie. The user can view a replay of notes as they appeared on the writing surface 105, or the user can view a replay of notes in the paper strip layout with visual indications of which paper strip corresponds to the current position of audio/visual playback. As another example, suppose a user viewed a webpage, an email, or a document on the computing device 115 while taking notes on the writing surface 105. The user may later review the webpage, email, or document while concurrently viewing taken notes. Snippets and paper strips having timestamps from the period the user reviewed the webpage, email, or document may be highlighted or contain some other visual indication of temporal correlation.
An example user interface of the smart pen computing system contains a “paper strip” display of content captured by the smart pen computing system. A paper strip contains one or more content items. Content items may include stroke data, clusters, snippets, character representations of stroke data, and linked data items such as contextual markers, commands, photographs, location information, audio recordings, video recordings, web pages, calendar entries, contact entries, emails, and documents. In an embodiment, a paper strip contains one or more clusters in a single snippet of text. In an alternate embodiment, a paper strip may contain clusters from multiple snippets, and clusters from a snippet may appear in multiple paper strips. The clusters may be displayed in a stroke data form (giving the appearance of handwriting) or character form (giving the appearance of typeset writing). Paper strips are oriented according to the direction in which stroke data in a snippet was written. For example, English writing is normally written left to right, so paper strips containing English writing are normally oriented horizontally.
In the paper strip interface, individual snippets are each treated as separate objects, each represented by a “paper strip” of the display. The term “paper strip” is used because the representation is analogous to cutting pages of a notebook into physical strips, each strip cut from one edge of the paper to the other in the direction of the writing (e.g., horizontally for English writing), and each strip including one sentence or idea (e.g., a snippet). These strips can then be collected from various pages or notebooks and sorted independently of their original position in the notebook. Similarly, the described paper strip interface may display snippets from multiple different pages or from multiple different writing surfaces 105. This enables the user to easily view and interact with individual snippets.
In one embodiment, the stroke data in a paper strip is arranged relative to the positioning of the corresponding handwriting on the writing surface 105. Furthermore, the relative positioning of stroke data with respect to at least one edge of the writing surface 105 may be preserved, although the relative positioning with respect to other edges of the writing surface 105 may be modified to improve presentation. For example, in English writing, where text is generally arranged in horizontal lines, the paper strip representation preserves the relative positioning of strokes to each other and with respect to the left and right edges of the writing surface so that these characteristics appear similar in the displayed paper strip as in the original writing. However, the vertical positioning of a snippet with respect to the top and bottom edges of a page that the snippet is written on may be disregarded in the paper strip presentation. Thus each paper strip appears as a strip bounded by the left and right edges of the writing surface 105 and upper and lower boundaries based on the height of the snippet. These strips are then arranged one under another in the display.
The first view 700A shows content items in the paper strips 705-735 at a default zoom (e.g., 100%). To increase the size of content items, a user initiates a zoom function. The zoom function applies a transformation such as scaling, shifting, rotating, and any combination thereof to content items in the paper strip interface. For example, if the computing device 115 has a touch screen display, a user may apply a multi-touch gesture (e.g., pinch to zoom) on a part of the screen to zoom in or out on content items and shift content the zoomed content towards the center of the screen.
The second view 700B shows the content items from the first view 700A after zooming in on the content item in paper strip 710A. This causes the displayed content to be scaled in size and may also cause the content to be shifted based on a specified zoom location. With the smart zoom feature, if scaling or shifting a content item would cause the content item to cross a left or right boundary of the visible display (for horizontal strips), then the content item is instead positioned just within the exceeded boundary. For example, the snippet in paper strip 705B is scaled up according to the amount calculated from the input gesture, but the snippet is shifted right to prevent the snippet from crossing the left boundary of paper strip 705B. The snippet in paper strip 710B is scaled up and is shifted left to prevent the snippet from crossing the right edge of paper strip 710B. Similarly, paper strips 715B and 720B contain snippets that have been scaled up and shifted left to prevent the snippets from crossing the left boundaries of their respective snippets. The paper strip 725B contains an image, which has been scaled up and shifted left to prevent the image from crossing the left edge of paper strip 725B. The paper strips 730B and 735B have been shifted out of the display area due to the increased vertical size of the paper strips 705B-725B, but these paper strips can be viewed by scrolling up or down in the vertical direction.
The third view 700C shows the content items from the second view 700B after zooming further in on the content of paper strip 710C. The clusters in paper strips 705C, 715C, and 720C have been scaled up in response to the scaling input until the left and right boundaries of the snippets coincide with the left and right boundaries of their respective paper strips. The resulting scaling in paper strips 705C, 715C, and 720C is less than that indicated by the input to zoom in so as to enable the content to remain within the visible display region. A process for applying the smart zoom technique is described in further detail below.
A paper strip layout is obtained 810, in one embodiment, by the processor 460. As described above, the paper strip layout comprises a plurality of paper strips, each paper strip containing a representation of a snippet and each paper strip occupying a strip (e.g., a horizontal strip) of the display screen. In an embodiment, the smart zoom method 800 is applied to a paper strip layout having at least two paper strips (e.g., a first and a second paper strip), which each have different content items (e.g., a first and a second content item). The content items may include representations of snippets, contextual data, or any other visual components depicted within a paper strip. The paper strip layout may have additional paper strips containing additional content items. Although the description below refers only to a first and second paper strip having first and second content items respectively, additional paper strips and additional content items may be treated similarly to the second content item by the smart zoom method 800.
An input is received 820 (e.g., from a user of the pen-based computing system 100) that requests a zoom level to apply to some selected content (e.g., first content) of a selected paper strip (e.g., a first paper strip). For example, referring to
The selected first content item is transformed based on the received input. Transforming the first content item includes applying 830 the first shifting component and the first scaling component to the first content item. For example, referring to the transition between
The content items in other paper strips may also be transformed based on the input. In one embodiment, content in other paper strips is zoomed in the same way as the selected item if applying this zoom will not cause any loss of visible content. For example, with respect to a second content item in a second (different) paper strip, it is determined 840 whether the second content item will remain within the boundaries of the visible display area if the same first transformation is applied to the second content item. In one embodiment, this determination involves checking the left and right boundaries of the display for horizontally oriented paper strips (e.g., for normal English writing). If the transformation would cause the paper strip to exceed upper and lower boundaries, the paper strip is simply enlarged to accommodate the vertical scaling. Paper strips may then be hidden from display when they are shifted across the upper and/or lower edges of a display, but can be viewed by scrolling the display up or down. This may be reversed for vertically oriented paper strips, used for example, in Japanese writing.
If applying the same first scaling component and first shifting component to the second content item keeps the second content item inside the boundary (e.g., the left and right boundaries of the visible display area of the second paper strip), then the first scaling and shifting components are applied 845 to the second content item to retain the same relative scale and position between the different content items in the different paper strips. Otherwise, a different transformation is applied to the second content item. This different transformation causes some change to the relative positioning and/or scaling between content items, but enables more content to remain visible on the zoomed in display. In one embodiment, the second transformation causes content in other paper strips to be scaled similarly to the selected first content where possible, but shifts the content horizontally as needed to enable it remain within the visible display region. For example, it is determined 850 if a different shift, applied with the first scaling, can keep the second content item inside the boundary. This is generally true whenever the width of the second content item, as scaled, is less than or equal to the width of the paper strip. If a different shift can be applied to keep the second content item within the boundary, then a second shift is calculated 860. In one embodiment, the second shift is applied such that second content item is just within the boundary that it would have otherwise exceeded (e.g., positioned a predefined distance from the boundary). In an embodiment, an overshoot distance is calculated from the distance by which the second content item would exceed the boundary of the second paper strip if the first scaling and first shift were applied. The second shift is then calculated 860 from the first shift modified by the overshoot distance applied in the opposite direction from the overshoot of the exceeded boundary. The calculated second shift is then applied 865 to the second content item along with the first scaling.
An example of this technique can be seen in the transition between
If it is determined 850 that a second shift cannot still keep the second content inside the boundaries of the second paper strip when the first scaling is applied, then a second scaling is calculated 870. This is generally the case when the width of the content will exceed the width of the paper strip if the first scaling is applied, and thus a reduced scaling is applied in order to enable the content to remain visible. In one embodiment, the maximum scaling is applied such that the content still remains within the boundaries when centered within the paper strip. This may also be determined based on a second scaling and second shift applied to the content. For example, in an embodiment, the second scaling is calculated 870 based on the distance between boundaries in the exceeded dimension and the width of the second content item in the exceeded dimension. For example, if paper strips are oriented horizontally, the second scaling may be calculated 870 based on the horizontal width between the left and right boundaries of the second paper strip, the horizontal width of the second content item, and the horizontal component of the first shift. The calculated second scaling is applied 875 along with the second shift. In an embodiment, the second shift may be determined to maximize the size of the second content item within the boundaries, to eliminate overlap between the second paper strip and an adjacent paper strip, or to eliminate overlap between content items in a paper strip.
An example of this approach can be seen in the transition between
In an embodiment, additional content items in additional paper strips may be present. These additional content items are treated individually similar to the second content item in the above process. For example, an additional content item is determined 840 to fit inside boundaries, and a transformation (e.g., 845, 865, 875) is applied to the additional content item as determined similar to the previously described method. In an alternate embodiment, a boundary other than the paper strip that contains a content item may be used to determine (e.g., 840, 850) if boundaries are exceeded or to calculate (e.g., 860, 870) transformations. For example, a paper strip boundary, a display boundary, a content item boundary, or a combination thereof may be used.
In an embodiment, the paper strip layout may be rotated, causing content items and paper strips to be rotated. The smart zoom process may be applied to content items in the paper strip responsive to a rotation of the paper strip layout. In an embodiment, paper strips are prepared for display after transforms are applied to the first and second content items (and any additional content items). In an embodiment, the paper strip layout is displayed on a computing device 115 or another device in the pen-based computing system 100. The entire paper strip layout or a portion thereof may be prepared for display.
The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.
Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a non-transitory computer-readable medium containing computer program instructions, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible computer readable storage medium, which includes any type of tangible media suitable for storing electronic instructions, and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
