Patent Grant
9224322
The present invention relates to methods and systems for conveying digital data. More specifically, the present invention relates to methods and systems for visually conveying digital data through video in an augmented reality environment.
Augmented reality, in general, involves augmenting one's view of and interaction with the real world environment with graphics, video, sound and/or other forms of computer-generated information. Augmented reality requires the use of an augmented reality device, which receives information from the physical, real world environment, processes the received information and, based on the processed information, presents the aforementioned graphics, video, sound and/or other computer-generated information in such a way that what the user experiences an integration of the physical, real world and the computer-generated information through the augmented reality device.
Often times, the physical, real world information received by the augmented reality device is only available over an active network connection, such as a cellular, WiFi, Bluetooth network or tethered Ethernet connection. If a network connection is not available, or use thereof is undesirable (for example, use of a network connection would be cost prohibitive), the augmented reality device will be unable to receive the physical, real world information and, in turn, unable to provide the user with the resulting video, sound and/or other computer-generated information necessary for the augmented reality experience.
There are, of course, other ways of conveying and receiving digital information. One such way is to convey and receive digital information visually. The general concept of visually conveying digital data is known. For example, Quick Response (QR) Codes are now widely used to visually convey digital information to a receiving device. QR Codes are commonly found on advertisements in magazines, on signs, on product packaging, on posters and the like. Typically, the receiving device, such as a smart phone, captures the QR code by scanning the QR Code using a camera application. The information contained in the QR Code, that is, the content of the code itself, may be almost anything. For instance, the content may be a link to a webpage, an image, a location or a discount coupon. One benefit of using a QR Code, or other like codes, is that the information is transferred immediately to the receiving device. The most significant benefit, however, is that the digital information can be conveyed to the receiving device visually, as it does not require a network connection.
It is therefore possible to visually convey physical, real world information, in digital format, to an augmented reality device, in the manner described above, that is, without a network connection. If the quantity of data required to support a given augmented reality application is relatively small, a code, such as a QR code or other like codes, may be used as described above. However, augmented reality applications often require a significant amount of data, or a constant stream of data, where the amount of data far exceeds that which can possibly be conveyed using a single QR or other like code.
A video or video related application for use in an augmented reality device is an example of an application that might require a significant amount of data, or a constant stream of data. For instance, the video or video related application might require the digital data so that the augmented reality device can generate and/or display, sound, graphics, text or other supplemental information relating to and synchronized with the real-world video presentation (e.g., a movie or television program) being viewed by the user. If a network connection is available, conveying the quantity of data or the constant stream of data required is not a problem. What is needed is a system and method for conveying this quantity of data, or the constant stream of data, to support a video or video related augmented reality application when a network connection is not available.
The present invention obviates the aforementioned deficiencies associated with conveying digital data associated with a video or video related application for an augmented reality device, where the digital data cannot be conveyed over a network connection because a network connection is either unavailable or, for any number of reasons, it is undesirable to do so. In general, the present invention achieves this by encoding the data, inserting the encoded data into the video, on a frame-by-frame or on predefined frames, and therefore conveying the data visually to the augmented reality device. The augmented reality device, upon receiving the data, can then use the data to supplement the video (e.g., a movie, video clip, television program) that the user is watching to augment and therefore enhance the user's experience.
One advantage of the present invention is that it permits the augmented reality device to receive digital data without the use of a network connection.
Another advantage of the present invention is that it allows for the conveyance of a significant quantity of data, or a constant stream of data, which may be required to supplement the video that the user is watching
Thus, in accordance with one aspect of the present invention, the above-identified and other advantages are achieved by a method of visually conveying digital data to an augmented reality device through video. The method involves inserting digital data into each of a plurality of video frames associated with the video. Accordingly, each of the plurality of video frames includes both video content and the inserted digital data. The method also involves displaying the video including each of the plurality of video frames such that the video including each of the plurality of video frames are available to be visually received by the augmented reality device, wherein the digital data represents data and/or information that supplements the video content.
In accordance with another aspect of the present invention, the above-identified and other advantages are achieved by a method of visually receiving digital data in an augmented reality device through video. The method involves visually capturing a plurality of video frames, wherein each of the plurality of video frames includes video content and digital data that has been inserted therein. The method also involves processing the digital data that was inserted into each of the plurality of visually received video frames and generating there from data and/or information that supplements the video content. The data and/or information that supplements the video content is then presented through the augmented reality device.
In accordance with still another aspect of the present invention, the above-identified and other advantages are achieved by an augmented reality device. The augmented reality device comprises a video sensor configured to visually capture video, wherein the video comprises a plurality of video frames, each including video content and digital data inserted therein. The augmented reality device also comprises a visual processor configured to process the digital data that was inserted into each of the plurality of visually received video frames and to generate there from data and/or information that supplements the video content. Still further, the augmented reality device comprises a rendering module configured to presenting, through the augmented reality device, the data and/or information that supplements the video content
Several figures are provided herein to further the explanation of the present invention. More specifically:
It is to be understood that both the foregoing general description and the following detailed description are exemplary. As such, the descriptions herein are not intended to limit the scope of the present invention. Instead, the scope of the present invention is governed by the scope of the appended claims.
In accordance with exemplary embodiments of the present invention, digital data is inserted into video (e.g., a movie, a video clip, a television program) and visually conveyed to and received by an augmented reality device. The augmented reality device, upon processing the visually conveyed digital data, can then supplement the video to enhance the user's viewing experience. For example, if the video is a movie, the digital data may be used by the augmented reality device to display subtitles in the user's desired language, or display additional video, graphics or text. It may also be used to generate sound to further enhance the user's experience.
Further in accordance with exemplary embodiments of the present invention, a portion of each of a number of video frames (e.g., each and every video frame) can be encoded with the aforementioned data that the augmented reality device will receive, through visual means, process and use to supplement or enhance the video that is being viewed by the user. For purposes of illustration, the digital data may be conveyed by inserting a QR code into each of the video frames. One skilled in the art will appreciate that a QR code has a maximum binary capacity of 2,953 bytes. Therefore, video displaying two QR codes at 30 frames per second can visually convey (not taking error correction into consideration) over 177 kilobytes of digital data per second to the augmented reality device. This is not intended to suggest that the present invention is limited to the insertion of only two QR codes into each video frame. The number of QR codes would likely depend on the resolution of the camera and the processing capabilities of the augmented reality device. The higher the resolution and the greater the processing capability, the greater the number of QR codes that may be inserted into each video frame. One skilled in the art will also appreciate the fact that an error correction scheme would likely be used to insure the integrity of the data being visually conveyed. However, even with an error correction scheme, the amount of data that can be visually conveyed to the augmented reality device is substantial.
As shown in
The augmented reality glasses 10 also include a Global Positioning System (GPS) unit 16. GPS units receive signals transmitted by a plurality of geosynchronous earth orbiting satellites in order to triangulate the location of the GPS unit. In more sophisticated systems, the GPS unit may repeatedly forward a location signal to an IMU to supplement the IMUs ability to compute position and velocity, thereby improving the accuracy of the IMU. In the present case, the augmented reality glasses may employ GPS to identify when the glasses are in a given location (e.g., a movie theater) where a video presentation having the inserted digital data is available. GPS units are also well known.
As mentioned above, the augmented reality glasses 10 include a number of features relating to sensory input and sensory output. Here, augmented reality glasses 10 include at least a front facing camera 18 to provide visual (e.g., video) input, a display (e.g., a translucent or a stereoscopic translucent display) 20 to provide a medium for displaying computer-generated information to the user, a microphone 22 to provide sound input and audio buds/speakers 24 to provide sound output. In a preferred embodiment of the present invention, the visually conveyed digital data would be received by the augmented reality glasses 10 through the front facing camera 18.
The augmented reality glasses 10 would likely have network communication capabilities, similar to conventional mobile devices, through the use of a cellular, WiFi, Bluetooth or tethered Ethernet connection. The augmented reality glasses 10 would likely have these capabilities despite the fact that the present invention provides for the visual conveyance and reception of digital data.
Of course, the augmented reality glasses 10 will also comprise an on-board microprocessor 28. The on-board microprocessor 28, in general, will control the aforementioned and other features associated with the augmented reality glasses 10. The on-board microprocessor 28 will, in turn, include certain hardware and software modules described in greater detail below.
Each of
The positioning of the digital data in the video frame is not essential to the present invention. However, it is preferable that the digital data be positioned such that the user, watching the video, cannot see or, at least, is not or less likely to be distracted by the presence of the digital data. In each of the three exemplary embodiments illustrated in
In still another exemplary embodiment as shown in
The bandwidth at which the digital data is visually conveyed also may vary. As mentioned above, absent any error correction scheme, the presentation of two different QR codes in each video frame, at 30 frames per second can visually convey over 177 kilobytes of digital data per second to the augmented reality device. Likewise, the bar codes and block codes illustrated in
The augmented reality shell 64, as shown, includes a number of components including a command processor 68, and environmental processor 72, a rendering services module 69 and a network interaction services module 70. It is will be understood that each of the functional modules and/or components may be hardware, software, firmware or a combination thereof. A brief description of each will now follow.
The environmental processor 72, in general, monitors the surrounding, real world environment of the augmented reality device based on input signals received and processed by the augmented reality device. The environmental processor 72 may be implemented, as shown in
The environmental processor 72, in turn, also includes several processing modules: a visual processing module 74, a geolocational processing module 78 and a positional processing module 80. The visual processing module 74 is primarily responsible for processing the received video, detecting and decoding the frames and processing the digital data included with the video that was visually conveyed to the augmented reality device.
The geolocational module 78 receives and processes signals relating to the location of the augmented reality mobile device. The signals may, for example, reflect GPS coordinates, the location of a WiFi hotspot, or the proximity to one or more local cell towers. As explained above, the geolocational processing module 78 may play a role in the present invention by notifying the augmented reality device when it is in a location where a video application may be used (e.g., a movie theater).
The positional processing module 80 receives and processes signals relating to the position, velocity, acceleration, direction and orientation of the augmented reality mobile device. The positional processing module 80 may receive these signals from an IMU (e.g., IMU 12). The positional processing module 80 may, alternatively or additionally, receive signals from a GPS receiver, where it is understood that the GPS receiver can only approximate position (and therefore velocity and acceleration) and where the positional processing module 80 can then provide a level of detail or accuracy based on the GPS approximated position. Thus, for example, the GPS receiver may be able to provide the general GPS coordinates of a movie theater, but the positional processing module 80 may be able to provide the user's orientation within the movie theater. The positional processing module 80 may be employed in conjunction with the visual processing module 74 to synchronize user head movements with viewing experiences (e.g., what the rendering services module 69 will render on the display and, therefore, what the user sees). Also, as stated above, the positional processing module 80 may be used to determine if and when the user has moved their head away from the video being presented, thus aiding in the determination whether and why synchronization has been lost (i.e., the augmented reality device is no longer receiving video and, more particularly, the digital data).
In addition to the environmental processor 72, the augmented reality shell 64 includes a command processor 68 and a rendering services module 69. The command processor 68 processes messaging between the modules and/or components. For example, after the visual processing module 74 processes the digital data that was visually received through the video, the visual processing module 74 communicates with the command processor 68 which, in turn, generates one or more commands to the rendering services module 69 to produce the computer-generated data (e.g., text, graphics, additional video, sound) that will be used to supplement the video and enhance the user's viewing experience.
The rendering services module 69. This module provides a means for processing the content of the digital data that was visually received and, based on instructions provided through the command processor 68, generate and present (e.g., display) data in the form of sound, graphics/animation, text, additional video and the like. The user can thus view the video and, in addition, experience the computer-generated information to supplement the video and enhance the viewing experience.
The general method 600 begins, of course, with the inclusion of digital data into a sequence of video frames associated with the corresponding video. This results in a video feed, as indicated by step 602, comprising a plurality of video frames, where each of the plurality of video frames includes the video content and the additional digital data that the augmented reality device will ultimately use to provide computer-generated data and/or information, supplement the video and enhance the user's viewing experience. It will be understood that the digital data may be included in each and every video frame or fewer than each and every video frame. As stated above, the amount of digital data that is visually conveyed may be limited by the bandwidth associated with the augmented reality device's camera and processing capabilities. It will also be understood that the manner in which the digital data is positioned within the video frame or integrated into the video content itself may vary, as explained above
The video feed may be displayed on anything from a television, a movie theater screen, a mobile device, a wall projection, or any other medium. Furthermore, the frame rate of the video is not particularly relevant here, nor are the dimensions of the medium on which the video is being displayed. The primary issue is that there is a series of encoded video frames, a plurality of which, include the additional digital data as explained above, which a video sensor associated with the augmented reality device can detect and pass to a frame processor, as explained herein below. Once the frame processor detects and stores the digital data, the system can process the data.
If a user is viewing the video with an augmented reality device, such as augmented reality device 10, a video sensor in the augmented reality device will capture the video and the digital data inserted therein, and convert the all of the received data back into a plurality of video frames for further processing, as indicated by step 604. In augmented reality device 10, the video sensor is the front facing camera 18.
As stated above, the captured video data, including the additional digital data, in the form of a plurality of video frames is passed on to a frame processor (not shown), as shown in step 606. The frame processor, in a preferred embodiment of the present invention, is implemented in the visual processing module 74. The primary function of the frame processor is to detect the presence of the digital data that is included with the video content, as shown by decision block 608. If, in accordance with the NO path out of decision block 608, the frame processor detects no digital data in a given video frame, the frame processor moves to the next frame and repeats the process. If, however, the frame processor does detect digital data in a given video frame, it will store the detected digital data, as shown in step 610. This is somewhat analogous to downloading data as the viewer is watching the video content. The frame processor then determines whether there are more video frames to analyze, as shown by decision step 612. If, in accordance with the YES path out of decision step 612, there are further video frames to analyze, the frame processor returns to step 606, and the method continues. If, instead, the frame processor determines there are no further video frames to analyze, in accordance with the NO path out of decision step 612, then all of the detected digital data will have been stored and the digital data can now be further analyzed, as shown by step 614, by the visual processing module 74. As explained above, the further analysis may involve determining the content of the digital data and, through the command processor 66, instruct the rendering services module 70 to provide computer-generated data and/or information in the form of text, graphics, animation, additional video, sound, to supplement the video and enhance the user's viewing experience.
In an alternative embodiment, the visual processing module 74 may further analyze the stored digital data as soon as the frame processor begins storing the digital data in memory. In other words, the frame processor may continue to analyze frames of video, detect any digital data contained therein, and store detected digital data while in parallel the other functions associated with the visual processing module 74 are analyzing digital data that has already been detected and stored by the frame processor.
With reference back to decision block 608, the frame processor may detect the presence of digital data through the use of markers. Such markers may, for example, be predefined data patterns or subtle color patterns. The markers may or may not be visible to the naked eye. However, the markers are recognizable by the frame processor. A marker may be included with the digital data at or near the edge or edges of the video frame or integrated into the video content itself, as explained above. Further, start and end markers may be employed, where the presence of an end marker would permit the frame processor to determine whether there is further digital data to detect and store, pursuant to decision step 612.
As mentioned previously, there are many possible applications for the present invention. To summarize, such applications may involve, for example, closed captioning, where the augmented reality device, such as augmented reality glasses 10, detects video frames that contain digital data reflecting closed captioning information that is ultimately displayed to the user while watching a television program or a movie. The application may involve subtitles that provide translation into a desired language or simply additional information that might be of interest to the user. The application may involve censorship, where the digital data may reflect information as to where the augmented reality device should place censor overlays on objectionable material. The application may involve intelligent advertising, where coupons and other items may be delivered or downloaded upon successful viewing of the advertisement video or by selecting an icon presented to the user through the display of the augmented reality device. And, as previously mentioned, the application may involve synchronized augmented reality movie content, wherein during a movie, additional content (e.g., in the form of additional and supplemental video, graphics and/or animation) may be displayed for the user in synchronicity with the video content, and wherein the additional content may or may not be restricted to the screen or viewing medium of the video. This last point is particularly significant as it distinguishes over present 3D techniques that are limited to presenting 3D content to the dimensions of the display screen or viewing medium. Thus, for example, a computer-generated image of a bird might appear to be flying around the room or theater because it is actually being projected on the display of the augmented reality device. The image would be unique to the perspective of that user based on the position of his or her head. This list of exemplary applications is not, however, intended to be limiting.
The present invention has been described above in terms of a preferred embodiment and one or more alternative embodiments. Moreover, various aspects of the present invention have been described. One of ordinary skill in the art should not interpret the various aspects or embodiments as limiting in any way, but as exemplary. Clearly, other embodiments are well within the scope of the present invention. The scope the present invention will instead be determined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4969041 | O'Grady et al. | Nov 1990 | A |
| 7814122 | Friedrich et al. | Oct 2010 | B2 |
| 8374383 | Long et al. | Feb 2013 | B2 |
| 8451266 | Hertenstein | May 2013 | B2 |
| 8913171 | Roberts et al. | Dec 2014 | B2 |
| 20070002077 | Gopalakrishnan | Jan 2007 | A1 |
| 20070242852 | Kumoluyi | Oct 2007 | A1 |
| 20080209062 | Barrett | Aug 2008 | A1 |
| 20100208033 | Edge et al. | Aug 2010 | A1 |
| 20110134108 | Hertenstein | Jun 2011 | A1 |
| 20110164163 | Billbrey et al. | Jul 2011 | A1 |
| 20110199479 | Waldman | Aug 2011 | A1 |
| 20120022924 | Runnels et al. | Jan 2012 | A1 |
| 20120069051 | Hagbi et al. | Mar 2012 | A1 |
| 20120206322 | Osterhout et al. | Aug 2012 | A1 |
| 20130147686 | Clavin et al. | Jun 2013 | A1 |
| 20140063055 | Osterhout et al. | Mar 2014 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20140035951 A1 | Feb 2014 | US |
