Paper notes have been broadly used in recording, sharing, and communicating ideas and information. For example, during a collaboration session (e.g., brainstorming session), participants write down ideas on repositionable paper notes, whiteboard, or paper, and then share with one another. In addition, people commonly use notes throughout the day to memorialize information or content which the individual does not want to forget. As additional examples, people frequently use notes as reminders of actions or events to take in the future, such as to make a telephone call, revise a document or to fill out a time sheet.
Software programs currently exist which permit computer users to create a software-based note in a digital form and to utilize the digital note within a computing environment. For example, a computer user may create a digital note and “attach” the digital note to an electronic document a desktop or electronic workspace presented by the computing environment.
Methods and systems for collaboration using notes allows users to capture physical notes via the users devices, such as a mobile phone with a digital camera, for conversion to corresponding digital notes and electronically transfer the digital notes to a shared screen or board during a video conferencing session other networked session, or between users devices.
The present disclosure describes techniques for creating and manipulating software notes representative of physical notes. For example, techniques are described for recognizing physical notes present within a physical environment, capturing information therefrom and creating corresponding digital representations of the physical notes, referred to herein as digital notes or software-based notes. Further, at least some aspects of the present disclosure are directed to techniques for managing multiple notes.
In general, notes can include physical notes and digital notes. Physical notes generally refer to objects with a general boundary and recognizable content. Physical notes can include the resulting objects after people write, draw, or enter via other type of inputs on the objects, for example, paper, white board, or other objects accepting the inputs. By way of examples, physical notes can include hand-written repositionable paper notes, paper, or film, white-board with drawings, posters, and signs. In some cases, physical notes can be generated using digital means, e.g., printing onto printable repositionable paper notes or printed document. In some cases, one object can include several notes. For example, several ideas can be written on a piece of poster paper or a white-board. Physical notes can be two-dimensional or three dimensional. Physical notes can have various shapes and sizes. For example, a physical note may be a 3 inches×3 inches note: a physical note may be a 26 inches×39 inches poster: and a physical note may be a triangular metal sign. In some cases, physical notes have known shapes and/or sizes. Digital notes generally refer to digital objects with information and/or ideas. Digital notes can be generated using digital inputs. Digital inputs can include, for example, keyboards, touch screens, digital cameras, digital recording devices, stylus, digital pens, or the like. In some cases, digital notes may be representative of physical notes.
In the example implementation, mobile device 15 includes, among other components, an image capture device 18 and a presentation device 28. In addition, although not shown in
In general, image capture device 18 is a camera or other component configured to capture image data representative of workspace 20 and notes 22 positioned therein. In other words, the image data captures a visual representation of an environment, such as workspace 20, having a plurality of visual notes. Although discussed as a camera of mobile device 15, image capture device 18 may comprise other components capable of capturing image data, such as a video recorder, an infrared camera, a CCD (Charge Coupled Device) array, a laser scanner, or the like. Moreover, the captured image data can include at least one of an image, a video, a sequence of images (i.e., multiple images taken within a time period and/or with an order), a collection of images, or the like, and the term input image is used herein to refer to the various example types of image data.
Presentation device 28 may include, but not limited to, an electronically addressable display, such as a liquid crystal display (LCD) or other type of display device for use with mobile device 28. In some implementations, mobile device 15 generates the content to display on presentation device 28 for the notes in a variety of formats, for example, a list, grouped in rows and/or column, a flow diagram, or the like. Mobile device 15 may, in some cases, communicate display information for presentation by other devices, such as a tablet computer, a projector, an electronic billboard or other external device.
As described herein, mobile device 15, and the software executing thereon, provide a platform for creating and manipulating digital notes representative of physical notes 22. For example, in general, mobile device 15 is configured to process image data produced by image capture device 18 to detect and recognize at least one of physical notes 22 positioned within workspace 20. In some examples, the mobile device 15 is configured to recognize note(s) by determining the general boundary of the note(s). After a note is recognized, mobile device 15 extracts the content of at least one of the one or more notes, where the content is the visual information of note 22.
In some example implementations, mobile device 15 provides functionality by which user 26 is able to export the digital notes to other systems, such as cloud-based repositories (e.g., cloud server 12) or other computing devices (e.g., computer system 14 or mobile device 16).
In the example of
In this example, mobile device 15 includes various hardware components that provide core functionality for operation of the device. For example, mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or data storage 68 such as static, random-access memory (SRAM) device or Flash memory device. I/O 76 may include one or more devices, such as a keyboard, camera button, power button, volume button, home button, back button, menu button, or presentation device 28 as described in
In general, operating system 64 executes on processor 70 and provides an operating environment for one or more user applications 77 (commonly referred to “apps”), including note management application 78. User applications 77 may, for example, comprise executable program code stored in computer-readable storage device (e.g., data storage 68) for execution by processor 70. As other examples, user applications 77 may comprise firmware or, in some examples, may be implemented in discrete logic.
In operation, mobile device 15 receives input image data and processes the input image data in accordance with the techniques described herein. For example, image capture device 18 may capture an input image of an environment having a plurality of notes, such as workspace 20 of
As shown in
In this example, note management application 78 includes image processing engine 82 that provides image processing and object recognition functionality. Image processing engine 82 may include image communication module 90, note identification module 86 and digital note generation module 88. In addition, image processing engine 82 includes image processing Application Programming Interfaces (APIs) 95 that provide a library of image manipulation functions, e.g., image thresholding, masking, filtering, edge detection, and the like, for use by the other components of image processing engine 82.
In general, image data may be stored in data storage device 68. In this example, note management application 78 stores images 97 within data storage device 68. Each of images 97 may comprise pixel data for environments having a plurality of physical images, such as workspace 20 of
As described herein, note identification module 86 processes images 97 and identifies (i.e., recognizes) the plurality of physical notes in the images. Digital note generation module 88 generates digital notes 99 corresponding to the physical notes recognized within the images 97. For example, each of digital notes 99 corresponds to one of the physical notes identified in an input image 97. During this process, digital note generation module 88 may update database 94 to include a record of the digital note, and may store within the database information (e.g., content) extracted from the input image within boundaries determined for the physical note as detected by note identification module 86. Moreover, digital note generation module 88 may store within database 94 metadata associating the digital notes into one or more groups of digital notes.
Further, note management application 78 may be configured, e.g., by user input 26, to specify rules 101 that trigger actions in response to detection of physical notes having certain characteristics. For example, user interface 98 may, based on the user input, map action to specific characteristics of notes. Note management application 78 may output user interface 98 by which the user is able to specify rules having actions, such as a note grouping action, or an action related to another software application executing on the mobile device, such as an action related to a calendaring application. For each rule, user interface 98 allows the user to define criteria for triggering the actions. During this configuration process, user interface 98 may prompt the user to capture image data representative of an example note for triggering an action and process the image data to extract characteristics, such as color or content. User interface 98 may then present the determined criteria to the user to aid in defining corresponding rules for the example note.
Image communication module 90 controls communication of image data between mobile device 15 and external devices, such as cloud server 12, computer system 14, mobile device 16, or image capture device 18. In some examples, image communication module 90 may, for example, allow a user to communicate processed or unprocessed images 97 of environments and/or digital notes and associated information extracted therefrom including metadata from database 68. In some examples, image communication module 90 exports this data to a zip file that may be communicated by FTP, HTTP, email, Bluetooth or other mechanism.
In the example of
In some example implementations, user interface 98 provides an image editor 96 that allows a user to edit the overlay image and/or the digital notes. In another example, digital note generation module 88 may include a process or processes that enhances the extracted information from the input image.
In some cases, the processing unit 110 can execute software or firmware stored in non-transitory computer-readable medium to implement various processes (e.g., recognize notes, extract notes, etc.) for the system 100A. The note content repository 140 may run on a single computer, a server, a storage device, a cloud server, or the like. In some other cases, the note content repository 140 may run on a series of networked computers, servers, or devices. In some implementations, the note content repository 140 includes tiers of data storage devices including local, regional, and central. The notes 120 can include physical notes arranged orderly or randomly in a collaboration space and the sensor 130 generates a visual representation of the notes 120 in the collaboration space.
In some implementations, the note recognition system 100A can include a presentation device (not shown in
In some embodiments, the note management system 100B can include one or more presentation devices 160 to show the content of the notes 120 to the user. The presentation device 160 can include, but not limited to, an electronically addressable display, such as a liquid crystal display (LCD), a tablet computer, a projector, an electronic billboard, a cellular phone, a laptop, or the like. In some implementations, the processing unit 110 generates the content to display on the presentation device 160 for the notes in a variety of formats, for example, a list, grouped in rows and/or column, a flow diagram, or the like.
Various components of the note recognition system and note management system, such as processing unit, image sensor, and note content repository, can communicate via a communication interface. The communication interface includes, but not limited to, any wired or wireless short-range and long-range communication interfaces. The short-range communication interfaces may be, for example, local area network (LAN), interfaces conforming to a known communications standard, such as Bluetooth standard, IEEE 802 standards (e.g., IEEE 802.11), a ZigBee or similar specification, such as those based on the IEEE 802.15.4 standard, or other public or proprietary wireless protocol. The long-range communication interfaces may be, for example, wide area network (WAN), cellular network interfaces, satellite communication interfaces, etc. The communication interface may be either within a private computer network, such as intranet, or on a public computer network, such as the internet.
A collaboration session allows each user to share digital notes within a group of participants.
As shown in
As shown in
The digital notes 205 can now be shared with other users during a video conferencing session other type of networked session, or between users devices. Examples of video conferencing applications include the Teams product by Microsoft Corporation and the Zoom product by Zoom Video Communications, Inc. Alternatively, the users can share screens or a board through a network connection that allows sharing or presenting a digital screen or a digital board to the other users through the network connection. The shared screen or board would be viewable by the users participating in the session. The video conferencing session, when used for this augmented reality feature, can include a default timer or a timer set by one of more of the users participating in the session.
As shown in
In order to transfer this digital note, a user holds mobile device 202 in view of digital camera 210 to position a target icon 212 at a particular location on screen or board 208. Moving mobile device 202 around in view of digital camera 210 causes corresponding movement of target icon 212 on shared screen or board 208. Although the target icon is shown as a circle with lines within the circle, it can be implemented with other types of icons or symbols.
As shown in
As shown in
As shown in
In this manner, as shown in
As another alternative, a user can transfer physical notes in addition to digital notes. By executing a particular command, a physical note can be automatically converted to a corresponding digital note and transferred to the shared screen or board. For example, a user could point to or circle a physical note, or press or activate a particular button on display device 203, when the physical note is within view of digital camera 201 in mobile device 202.
This augmented reality feature thus provides a user with a way to capture physical notes and a private area to work on the corresponding digital notes, as shown by notes 205 and
When collaborating with other users or participants in shared boards, users can capture physical notes converted to digital notes and create digital notes to a private area of an electronic device (e.g., mobile phone) where the digital notes can be sorted, edited, and deleted before sharing with other users or participants. This feature captures the important aspect of analog collaboration where users or participants work “alone together,” privately capturing and creating their own notes and then selectively sharing those digital notes with the other users or participants. This feature also democratizes thought since the participants do not view everyone's notes at the same time, do not feel the pressure to conform to the notes of others, and when digital notes are finally moved into the shared boards, the digital notes all appear the same. These aspects are an inherent strength of physical repositionable notes, and this feature provides for that advantage with digital notes.
Notes can be sorted and/or visualized many ways in the same board. For instance, users or participants could switch between the following: displaying digital notes in visual boards that display a collection of notes: displaying an hierarchical list that shows only the text of notes as converted through optical character recognition: and a KANBAN board (workflow visualization tool) that shows notes visually, but removes spatial information to only focus on state and sorting. Users or participants could also sort digital notes by organizing and/or filtering them by known attributes (metadata) such as timestamp, color, size, alphanumerical text, author, session ID, or other attributes. This feature provides new capabilities to digital notes that are not possible in the analog world of physical (e.g., paper) repositionable notes.
This feature involves finding coherence or meaning within all the notes. For instance, this feature could involve using a computer to help find common concepts, automatically create groups of digital notes based on the content of those concepts, create groups and then automatically sort the digital notes based on content, automatically find duplicates, or automatically perform other actions on the notes. This feature can be powered by machine learning techniques. For example, the system could provide predefined electronic templates (e.g., flowcharts, themes, swot, fishbone, etc.) and utilize machine learning to assist the user by automatically pre-sorting the digital notes into the selected template logically based on content.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/IB2022/053279 | 4/7/2022 | WO |
Number | Date | Country | |
---|---|---|---|
63182060 | Apr 2021 | US |