Patent Application
20200118343
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
This invention relates generally to augmented reality (AR), and, more particularly, to methods, systems and devices supporting real-time interactions in AR environments.
The present invention is specified in the claims as well as in the below description. Preferred embodiments are particularly specified in the dependent claims and the description of various embodiments.
A skilled reader will understand, that any method described above or below and/or claimed and described as a sequence of steps or acts is not restrictive in the sense of the order of steps or acts.
Below is a list of method or process embodiments. Those will be indicated with a letter “M”. Whenever such embodiments are referred to, this will be done by referring to “M” embodiments.
The above features along with additional details of the invention, are described further in the examples herein, which are intended to further illustrate the invention but are not intended to limit its scope in any way.
Objects, features, and characteristics of the present invention as well as the methods of operation and functions of the related elements of structure, and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification.
As used herein, unless used otherwise, the following terms or abbreviations have the following meanings:
“2D” or “2-D” means two-dimensional;
“3D” or “3-D” means three-dimensional;
“AR” means augmented reality.
“VR” means virtual reality.
A “mechanism” refers to any device(s), process(es), routine(s), service(s), or combination thereof. A mechanism may be implemented in hardware, software, firmware, using a special-purpose device, or any combination thereof. A mechanism may be integrated into a single device or it may be distributed over multiple devices. The various components of a mechanism may be co-located or distributed. The mechanism may be formed from other mechanisms. In general, as used herein, the term “mechanism” may thus be considered to be shorthand for the term device(s) and/or process(es) and/or service(s).
In the following, exemplary embodiments of the invention will be described, referring to the figures. These examples are provided to provide further understanding of the invention, without limiting its scope.
In the following description, a series of features and/or steps are described. The skilled person will appreciate that unless required by the context, the order of features and steps is not critical for the resulting configuration and its effect. Further, it will be apparent to the skilled person that irrespective of the order of features and steps, the presence or absence of time delay between steps, can be present between some or all of the described steps.
It will be appreciated that variations to the foregoing embodiments of the invention can be made while still falling within the scope of the invention. Alternative features serving the same, equivalent or similar purpose can replace features disclosed in the specification, unless stated otherwise. Thus, unless stated otherwise, each feature disclosed represents one example of a generic series of equivalent or similar features.
Devices
Smartphones and other such portable computing devices are ubiquitous, and much of today's communication takes place via such devices. With current devices, users may interact and converse in real time using various computer and/or telephone networks. In addition, many young children use such devices for reading, playing games, and sometimes even communication.
Such devices may be used to experience augmented reality (AR) environments such as those described herein. Accordingly, for the purpose of this specification, we first describe some standard functionalities of a typical device 100 such as a smartphone or tablet computer (e.g., an iPhone or Android phone, or an iPad, or the like). This will be described with reference to
As should be appreciated, the drawings in
In conventional usage, when the front camera 104 is active, then the display screen 102 shows an image corresponding to the view of the front camera 104 (e.g., in the example of
As used herein, the term “virtual object” or (“object” in the context of a virtual space) refers to any object or part thereof, real or imaginary, and may include faces, bodies, such as an avatar or the like. A virtual object may be static or dynamic and may be animatable and otherwise manipulatable in the object's virtual space. A virtual object may be associated in the AR space with one or more other objects, including other virtual and real-world objects. For example, a virtual object may be a face associated with a real person or animal in an AR space.
With reference now to
Device 200 may include one or more processors 202, display 204 (corresponding, e.g., to screen 102 of device 100), and memory 206. Various programs (including, e.g., the device's operating system as well as so-called applications or apps) may be stored in the memory 206 for execution by the processor(s) 202 on the device 200.
The memory may include random access memory (RAM), caches, read only storage (e.g., ROMs, etc.). As should be appreciated, the device 200 (even if in the form of a smartphone or the like) is essentially a computing device (described in greater detail below).
The device 200 may include at least one camera 208, preferably including one or more front cameras 210, and one or more rear cameras 212. The cameras may be capable of capturing real time view images (still or video) of objects in their respective fields of view. In some embodiments hereof, the front and rear cameras may operate at the same time (i.e., both the front and rear cameras can capture images at the same time). That is, in some embodiments, the front camera(s) 210 may capture video or still images from the front of the device while, at the same time, the rear camera(s) 212 may capture video or still images from the rear of the device. Whether and how any of the captured images get displayed, rendered or otherwise used is described below. The front cameras 210 may correspond to front camera(s) 104 in device 100, and the rear cameras 212 may correspond to the rear camera(s) 108 in device 100.
The memory 206 may include camera memory 218 provided or allocated for specific use by the cameras. The camera memory 218 may be special purpose high-speed memory 208 (e.g., high-speed frame buffer memory or the like) and may include front camera memory 220 for use by the front camera(s) 210, and rear camera memory 222 for use by the rear camera(s) 212.
The device 200 may also include one or more microphones 224 to pick up sound around the device and one or more speakers 226 to play audio sound on the device. The device may also support connection (e.g., wireless, such as Bluetooth, or wired, via jacks) of external microphones and speakers (e.g., integrated into a headset).
The device may include one or more sensors 228 (e.g., accelerometers, gyroscopes, etc.) and an autonomous geo-spatial positioning module 229 to determine conditions of the device such as movement, orientation, location, etc. The geo-spatial positioning module 229 may access one or more satellite systems that provide autonomous geo-spatial positioning, and may include, e.g., the GPS (Global Positioning System), GLONASS, Galileo, Beidou, and other regional systems.
The device preferably includes one or more communications mechanisms 230, supporting, e.g., cellular, WiFi, Bluetooth and other communications protocols. For example, if the device 200 is a cell phone, then the communications mechanisms 230 may include multiple protocol-specific chips or the like supporting various cellular protocols. In this manner, as is known, the device may communicate with other devices via one or more networks (e.g., via the Internet, a cellular network, a LAN, a WAN, a satellite connection, etc.).
In some exemplary embodiments, devices may communicate directly with each other, e.g., using an RF (radio frequency) protocol such as WiFi, Bluetooth, Zigbee, or the like.
The communications mechanisms 230 may also support connection of wireless devices such as speakers and microphones mentioned above.
The AR App
In some aspects, exemplary embodiments hereof provide a system that creates, supports, maintains, implements and generally operates various augmented reality (AR) elements, components, collaborations, interactions, experiences and environments. The system may use one or more devices such as the device 200 as a general AR processing and viewing device, and as such, the system may include an AR mechanism that may reside and operate on the device 200 as depicted in
As depicted in
The AR APP 232 may include one or more of the following mechanisms:
1. Augmenter mechanism(s) 234
2. Face recognition mechanism(s) 236
3. Facial manipulation mechanism(s) 238
4. Communication mechanism(s) 240
5. Animation mechanism(s) 242
6. 2-D and 3-D modeling mechanism(s) 244
7. Speech or voice manipulation mechanism(s) 246
8. Speech or voice augmentation mechanism(s) 248
9. Voice Recognition mechanism(s) 250
10. Gesture recognition mechanism(s) 252
11. Gesture manipulation mechanism(s) 254
This list of mechanisms is exemplary, and is not intended to limit the scope of the invention in any way. Those of ordinary skill in the art will appreciate and understand, upon reading this description, that the AR App 232 may include any other types of recognition mechanisms, augmenter mechanisms, manipulation mechanisms, and/or general or other capabilities that may be required for the AR App 232 to generally perform its functionalities as described in this specification. In addition, as should be appreciated, embodiments or implementations of the AR App 232 need not include all of the mechanisms listed, and that some or all of the mechanisms may be optional.
The mechanisms are enumerated above to provide a logical description herein. Those of ordinary skill in the art will appreciate and understand, upon reading this description, that different and/or other logical organizations of the mechanisms may be used and are contemplated herein. It should also be appreciated that, while shown as separate mechanisms, various of the mechanisms may be implemented together (e.g., in the same hardware and/or software).
As should be appreciated, the drawing in
In operation, the AR App 232 may use each mechanism individually or in combination with other mechanisms. When not in use, a particular mechanism may remain idle until such time its functionality may be required by the AR App 232. Then, when the AR App 232 may require its functionality, the AR App 232 may engage or invoke the mechanism accordingly.
Note also that it may not be necessary for the AR App 232 to include all of the mechanisms listed above. In addition, the AR App 232 may include mechanisms that may not be in the above list.
The different mechanisms may be used for different types of AR experiences or programs that the AR App 232 may provide. In addition, the end user may desire to run and experience a specific AR program(s), and accordingly, may only require the mechanisms of the AR App 232 that may drive that particular AR program(s). In this situation, the unused mechanisms (if included in the AR App 232) may sit idle, the AR App 232 may not include the unnecessary mechanism, or any combination thereof.
The AR App 232 may orchestrate the use of various mechanisms combined with native functionalities of the device 200 to perform, create, maintain and generally operate a wide variety of different types of AR experiences, interactions, collaborations and environments.
As noted above, a virtual object may be animatable and otherwise manipulatable in the object's virtual space. Such animation and/or manipulation may use various of the AR app's mechanisms, including, e.g., augmenter mechanism(s) 234, facial manipulation mechanism(s) 238, animation mechanism(s) 242, 2-D and 3-D modeling mechanism(s) 244, and gesture manipulation mechanism(s) 252.
Speech produced by an object in an AR space (virtual or real, including a real person) may be manipulated or augmented by various of the AR app's mechanisms, including, e.g., speech or voice manipulation mechanism(s) 246, and speech or voice augmentation mechanism(s) 248. As used herein, the term “speech,” in the context of an AR environment or space, refers to any sound that may be used to represent speech of an object or real-world person. Augmented or manipulated speech may not correspond to any actual speech or language, and may be incomprehensible.
The Backend Platform
As should be appreciated, although the functionality provided by the AR app 232 is preferably available on the device 200, aspects of the functionality may be provided or supplemented by mechanisms located elsewhere (e.g., a backend platform).
Accordingly, as shown, for example, in
Data from the devices 258, 260, etc. may be communicated to each other as well as to the backend platform 256. The backend platform 256 may include one or more servers that may include CPUs, memory, software, operating systems, firmware, network cards and any other elements and/or components that may be required to the backend platform 256 to perform its functionalities.
Embodiments or implementations of backend platform 256 may include some or all of the functionalities, software, algorithms and mechanisms necessary to correlate, process and otherwise use all of the data from the devices 258, 260, etc.
A backend platform 256 according to exemplary embodiments hereof may include services or mechanisms to support one or more of the following: 2-D and 3-D modeling mechanism(s) 264, facial manipulation/animation mechanism(s) 266, animation mechanism(s) 268, face recognition mechanism(s) 270, voice manipulation mechanism(s) 272, gesture recognition mechanism(s) 274, speech and/or voice recognition mechanisms 276, speech and/or voice augmentation mechanism(s) 278, language translation mechanism(s) 280, voice-to-text mapping mechanism(s) 282, etc. In general, there may be one or more mechanisms on the backend platform 256 corresponding to each of the device mechanisms (in the AR app 232).
A particular implementation of a backend platform may not have all of these mechanisms and may include other mechanisms not listed here. The mechanisms on the backend platform may augment or replace mechanisms on the devices, e.g., on an as-needed basis.
Thus, a device, while in conversation with another device, may also be connected to a backend platform for support with one or more of its AR mechanisms.
Although various mechanisms are shown on the backend platform 256 in
As an example, a backend platform may support the 2-D and 3-D modeling mechanism(s) 244 on a device (e.g., one of devices 258, 260, etc.). As should be appreciated, building and sharing comprehensive 3-D models of augmented physical environments may require significant processing power as well as a large amount of memory—sometimes more than is available on a single device.
Data from one or more devices may be communicated to/from the backend platform 256 on a continual basis, and 2-D and 3-D modeling mechanism(s) 264 on the backend platform 256 may accordingly create 2-D or 3-D models and communicate them back to the devices. Some of the processing of the data and model creation may occur on the devices and some of the processing and model creation may occur on the backend platform 256, or any combination thereof
Real Time
Those of ordinary skill in the art will realize and understand, upon reading this description, that, as used herein, the term “real time” means near real time or sufficiently real time. It should be appreciated that there are inherent delays in electronic components and in network-based communication (e.g., based on network traffic and distances), and these delays may cause delays in data reaching various components. Inherent delays in the system do not change the real time nature of the data. In some cases, the term “real time data” may refer to data obtained in sufficient time to make the data useful for its intended purpose.
Although the term “real time” may be used here, it should be appreciated that the system is not limited by this term or by how much time is actually taken. In some cases, real-time processing or computation may refer to an online process or computation, i.e., a process or computation that produces its answer(s) as data arrive, and generally keeps up with continuously arriving data. The term “online” computation is compared to an “offline” or “batch” computation.
Operation of aspects of the AR system, including AR App 232, alone or in conjunction with a backend platform 256, will be described by way of several detailed examples.
The examples provided below are chosen to illustrate different types or combinations of AR experiences and programs that exemplary implementations of the AR App 232 may execute. Each example may purposely demonstrate the utilization of different mechanisms (or combinations of different mechanisms) within the AR App 232. Those of ordinary skill in the art will appreciate and understand, upon reading this description, that the examples are not limiting and that the AR App 232 may be used in different way.
For example, one descriptive example presented below may include an AR program that may include two or more users each with a device 200 that may be running the AR App 232. The users may be in communication with each other via the devices 200 and may each view their respective AR environments augmented with images of the other user. Note that this example may demonstrate the utilization of the communication mechanism 240, the augmenter mechanism 234, the facial recognition mechanism 236, the facial manipulation mechanism 238, the voice manipulation mechanism 246 and the animation mechanism 242. The functionalities of the above mechanisms may be used independently or in any combination with one another as necessary for the AR App 232 to perform the functionalities as described.
Another descriptive example presented below may include a storytelling program that may augment the viewed environment of the user with virtual objects or elements as a part of a moving storyline. Note that this example may demonstrate the utilization of the communication mechanism 240, the augmenter mechanism 234, the gesture recognition mechanism 252 and the voice recognition mechanism 250. The functionalities of the above mechanisms may be used independently or in any combination with one another as necessary for the AR App 232 to perform the functionalities as described.
Note that some of the examples presented may use similar but not identical combinations of the mechanisms (i.e., the examples may share the use of some of the mechanisms but not others). Note also that the AR App 232 may include some mechanisms that may not be necessarily used in some of the examples, and that these mechanisms (if implemented in a particular embodiment or implementation) may rest idle when not in use. Alternatively, the unused mechanisms may not be included in AR App 232 (e.g., per a user's decision during the configuration of the AR App 232).
It should be appreciated that the examples presented do not limit the scope of the current invention to the specific functionalities that they may demonstrate, and that other mechanisms and/or combinations of mechanisms included with and engaged by the AR App 232 may result in other functionalities or combinations of functionalities that may not specifically be demonstrated in the examples but that are within the scope of the invention.
Exemplary embodiments hereof (e.g., the AR app 232, alone or in conjunction with a backend platform 256) may be used for real-time communication and collaboration with AR.
For example, as depicted in
Although the term “collaborative” may be used throughout this specification to describe the operation of the device 200 and the AR program 232, it should be appreciated that the invention is not limited by the term “collaborative,” and the term encompasses any kind of conversation or interaction between two or more users, including, without limitation, real-time voice and video conversations and/or interactions and/or chatting.
Using the AR program 232, a device's front and rear cameras are simultaneously active, and the first user's device 300 renders on its display (i) a view from its rear camera 308, augmented with (ii) one or more images based on the view from user U2's front camera 304′. At the same time, the second user's device renders on its display (i) a view from its rear camera 308′, augmented with (ii) one or more images based on the view from user U1's front camera 304.
As noted, using the camera(s) 208, the device 200 can receive live video of a real-world, physical environment. As used herein, the term “live” means at the time or in substantially real time. Thus, “live video” corresponds to video of events, people, places, etc. occurring at the time the video is transmitted and received (taking into account processing and transmission delays). Similarly, “live audio” refers to audio captured and transmitted at the time of its capture.
Thus, each user's display shows a real time view from their rear camera, augmented with an image based on a real time view of the other user's front camera. As explained, the images that a device renders from another user's device are based on the images from that other device, but need not necessarily be the same as the images from the other device. For example, the images may be stylized, augmented, animated, resized, etc.
In a simple case, the images from the front camera(s) of the first user's device are superimposed in some manner (on the display of the second user's device) on the images from the rear camera(s) of the second user's device, and vice versa.
Communication and collaboration via the AR app is not limited to two users.
An example AR collaboration or interaction with three users is shown in the stylized drawing in
In general, a user's screen may display images from any of the other user's camera(s), alone, or along with AR images.
With reference again to
An augmenter mechanism 234 combines real-time video images from the device's rear camera(s) 212 (e.g., from rear camera memory 222) with some form of the image(s) received in real-time from one or more other devices. The images from the other devices may be augmented.
In some preferred embodiments, the AR app 232 has (or has access to) one or more mechanisms, including: face recognition mechanism(s) 236, facial manipulation mechanism(s) 238, and voice manipulation mechanism(s) 246. Each device 200 may use one or more of these mechanisms to manipulate the images and/or audio from its front camera(s) 210 prior to sending the images and/or audio to other users. For example, using the AR app 232, a device 200 may use the face recognition mechanism(s) 236 to find and store into memory 206 the user's face in the images received in real time by its front camera 210. The AR app 232 may the use facial manipulation mechanism(s) 238 to manipulate images (e.g., stored images) of the user's face prior to sending them to other users in a conversation or collaboration. The facial manipulation may, e.g., correspond to the user's facial expressions and/or gestures. The other users will then receive manipulated facial images or a stylized version of the sending user's face. The user may thus, e.g., send a stylized or graphically animated version of their face that moves in real time in correspondence to their actual face in front of the camera. The receiving user then receives the stylized or graphically animated version of the other user's face and uses their augmenter mechanism 234 to create an AR image combining the view from their rear camera and the received image.
In some embodiments, users may manipulate their voices using voice manipulation mechanism(s) 246, so that the audio that is sent to other users is a modified version of their actual voice. The voice manipulation occurs in real time and may use various kinds of filters and effects. Thus, the receiving user may receive a modified audio from a sending user. The voice manipulation may simply modulate aspects of the captured audio or it may completely change the audio (e.g., provide a translation to another language).
As should be appreciated, the audio and video manipulations may occur at the same time.
The augmenter mechanism 234 may superimpose some form of the image(s) received in real-time from one or more other users, or it may manipulate or augment them. Preferably, the augmenter mechanism 234 manipulates the images and/or audio received from other users before combining the received images and/or audio with the video from its own rear camera. Thus, in some preferred embodiments, preferably the combined and rendered image includes a manipulation of the images received from the other user's devices.
The AR app 232 may include one or more animation mechanisms 242, and, in some embodiments, the augmenter mechanism 234 may use the animation mechanism(s) 242 to animate images received from other users. For example, when the received image is a user's face (possibly modified and/or manipulated by the sender), the receiving device may put that face on a frame that may be animated (e.g., an avatar or the like) and then animate the combined face and frame using the facial manipulation mechanism(s) 238 and animation mechanism(s) 242. In these cases, the receiver's display will show the image from their rear camera, augmented with an animated version of the received image(s). The animated images may be in 2-D and/or 3-D.
Thus, e.g., with reference again to
For the sake of this description, it can be assumed that all users in an AR collaboration are using a device running the AR app 232 with at least some of the functionality of the app. As should be appreciated, there is no requirement that the devices be the same or that all versions of the app support all of the features or that all supported features are enabled or being used. For example, a user may choose not to use the voice manipulation mechanism 246. As another example, a device may not support the simultaneous use of front and back cameras, in which case, that device may not have an AR view that other devices in the collaboration can create.
As shown in
Aspects of the augmenting, including the animation, may take place in part or entirely on the device. Additionally, the device may use aspects of the back-end platform's facial manipulation/animation mechanism(s) 266.
In some exemplary embodiments, a device's augmenter mechanism(s) 234 may add additional information or content to the image rendered. For example, as shown in
Similarly, device 602 (also an embodiment of device 200), may render object(s) 606 and/or text 610. Note that the objects and/or text rendered on the various devices need not be the same.
In summary, in some aspects, with reference again to the drawing in
Similarly, with reference again to the drawing in
In another example of the AR program 232 to further illustrate aspects of its functionality, consider the example shown in
Using the AR program 232, a device's front and rear cameras are simultaneously active, and the user's device 700 renders on its display (i) a view from its rear camera(s) 708, augmented with (ii) one or more AR images based on image(s) captured by the device's front camera(s) 704. In a preferred embodiment, the user's face (based on real time images captured by the front camera(s) 704, and using face recognition mechanism(s) 236) is mapped to a part of the images rendered on the display of device 700. In the example shown in
The image rendered on the device's display may include other AR images or objects that are not based on either the front or rear camera views. For example, as shown in
In another example, as shown in the stylized drawing in
Since the images captured by the front and rear cameras are live and in real time, changes to either camera's view will cause corresponding changes to the rendered AR view. For example, if the user smiles then the AR facial image of the user may also smile. If the user changes the real-world view of either camera, then the corresponding rendered images will change. For example, if the user faces the rear camera to another location then the rendered AR image will include that other location.
With reference again to
The augmenter mechanism(s) 234 may combine real-time video images from the devices rear camera(s) 708 (e.g., from rear camera memory 222) with some form of the image(s) received in real-time from the device's front camera(s). In some embodiments, the device may also include images (real and/or augmented) from one or more other users.
In some preferred embodiments, the AR app 232 may use its face recognition mechanism(s) 236 and facial manipulation mechanism(s) 238, to manipulate the images from its front camera(s) 704. For example, using the AR app 232, a device 700 may use the face recognition mechanism(s) 236 to find the user's face in the images received in real time by its front camera 704. The app 232 may the use the facial manipulation mechanism(s) 238 to manipulate the images of the user's face prior to rendering them as AR images. The facial manipulation may, e.g., correspond to the user's facial expressions and/or gestures. The user may thus, e.g., render a stylized or graphically animated version of their face that moves in real time in correspondence to their actual face in front of the camera in order to create an AR image combining the view from their front and rear cameras.
The augmenter mechanism 234 may simply superimpose some form of the image(s) received in real-time from the front camera(s), or it may manipulate or augment them. Preferably the augmenter mechanism 234 manipulates the images received from the front camera(s) before combining them with the video from its rear camera.
In some embodiments, the augmenter mechanism 234 may use the animation mechanism(s) 242 to animate AR images. For example, when the image from a front camera is a user's face (possibly modified and/or manipulated), the device may put that face on an animatable frame (e.g., an avatar or the like) and then animate the combined face and frame using the animation mechanism(s) 242. In these cases, the display will show the image from their rear camera, augmented with an animated version of the image(s) from the front camera. The animated images may be in 2-D and/or 3-D.
Thus, e.g., with reference again to
The augmenter mechanism(s) 234 may, optionally, add a received video to a body, and then animate the combined body and face (using, the animation mechanism(s) 242). The augmented part (based on the video signal from the front camera(s)) is combined (at 908) with the captured video from the rear camera(s) and then rendered (at 910).
In some exemplary embodiments, the augmenter mechanism 234 may add additional information or content to the image rendered. For example, the device (an embodiment of device 200 of
Unified Viewing Space
In general, each device has a view of the same unified space. Thus, as shown in logical diagram
The virtual or AR objects 916 in the viewing space 912 may depend on the application and/or use case. For example, in the example in
The virtual/AR objects 916 may be generated by the application on one or more of the devices or by another application or the backend.
Those of ordinary skill in the art will appreciate and understand, upon reading this description, that the various devices D1 . . . Dn need not be in the same physical (or geographic) location.
In another exemplary embodiment, the AR program 232 may function for or in support of storytelling. For this embodiment, the AR program is also referred to as the AR Story program 232.
As an example of the story telling functionality, with reference to the drawing in
If the device also has a front camera (facing the user), then that front camera may capture images of the user at the same time that the rear camera is capturing the images of the bed. As explained below, images of the user from the front camera may be used for AR and/or control of the AR Story program 232.
When the AR story program 232 is used, the user may select a story, after which information about the story is loaded into the device's story memory.
As shown, e.g., in
As used herein, the term “story” has its broadest possible meaning, including, without limitation, an account and/or representation of real and/or imaginary events and/or situations and/or people and/or information and/or things. The scope of the invention is not limited by the nature of any story or by its relation to any real or fictitious events or people.
At each event, the AR story program 232 performs the event actions associated with that event. These event actions may include audio actions, text actions, AR items, and transition information or other types of event actions. That is, the event actions may include one or more of: (i) render one or more sounds; (ii) displaying text; and/or (iii) rendering one or more AR items. Other types of event actions may also be included. An event may also provide a transition rule or description that defines the manner in which the next event is started or reached. For example, an event may flow automatically to the next event after a preset period of time (e.g., 15 seconds), or an event may require some form of user interaction (e.g., one or more of: voice command, touch, facial gesture, hand gesture, arm gesture, finger gesture, body gesture, etc.) to go to the next event. These manners of transition or flow control are given only by way of example, and those of ordinary skill in the art will appreciate and understand, upon reading this description, that different and/or other flow controls may be used. The term “voice command” may refer to any sound expected by the AR story program 232, and may include text read by the user. The AR story program 232 may use the text, for example, the text of a story being read by a user may be used to trigger a transition to another event (usually, but not necessarily, the next event in the list).
A story 1100 may transition from one event to another event at different junctions within the story 1100. For instance, an event transition may occur when a new event action is to be performed, such as when new text is to be displayed on the screen, when a new character enters the story, when the setting changes or at any other juncture along the story that may require a new event and new event actions.
In another example, the AR story program 232 may use the front camera 1002 of the exemplary device 1000 to capture gestures (including facial gestures) made by the user during the event flow. As mentioned above, front camera(s) 1002 and rear camera(s) 1008 may both be active simultaneously, with rear camera(s) 1008 capturing the real-world images that may be augmented, and with front camera(s) 1002 capturing images of the user (e.g. gestures that the user may perform). For instance, the storyline may include a character asking the user to point to which road the character should take along a landscape, the left road or the right road. If the user points to the left, the gesture recognition mechanism 252 of the story program 232 may use front camera(s) 1002 to capture and recognize the left-pointing gesture as a transition rule trigger and AR story program 232 may transition to the next event that may involve the character taking the road to the left. If, however the user points to the right, the gesture recognition mechanism 252 of the story program 232 may capture and recognize the right-pointing gesture as a transition rule trigger and AR story program 232 may transition to the next event that may involve the character taking the road to the right. The gesture recognition mechanism(s) 252 may also capture and/or recognize other types of gestures such as a smile that may cross the user's face, a thumbs-up gesture, or any other type of gesture that may be captured and recognized by the AR story program 232 and/or device 1000 as a transition rule or trigger.
It should be noted that in the above example the user controlled the order of events by choosing which road for the character to take (i.e. the left road or the right road). In this case, the sequence of events of story 1100 may not comprise only a linear sequence of events as depicted in the flowchart of
For each audio item in an event, the event may specify when the audio is played (e.g., at the start of the event, repeated during the event, etc.). For each text item, the event may specify when and where (on the screen) the text is displayed (e.g., location, format, duration). For each AR item, the event information may specify how and where it is to be displayed and whether or not it is to be animated (e.g., location, format, duration, animation).
Going back to the example of
Next, the user may trigger an event transition rule (e.g. via a voice command, a gesture, or other interaction), and the story 1100 may transition to a new event depending on the trigger interaction. For example, the story 1100 may transition to the event depicted in
The AR Story App 232 may use the microphone(s) 224 and speech or voice recognition mechanism(s) 250 on the device 200 in order to determine whether a particular phrase has been spoken by the user.
As noted above, if the device's camera is moved, then the displayed scene, including the AR aspects thereof, will change, preferably in real time. Thus, for example, with the scene depicted in
The story may be narrated, at least in part, by a recording (which would be included in the audio part of the event actions), or it may be read or spoken by someone in real time. For example, a parent may read the story (from an external source, e.g., a book or web site or the like) while the AR story App 232 is running. The narrator may pause at times and require interaction (e.g., by touch or voice) before continuing. This approach allows a reader to synch up with the AR animation/activity on the screen (and vice versa). The narrator/reader need not be co-located with the user/device. For example, a parent may read a story to a child over the telephone, with the device running the story app to present the AR version of the story.
In addition, the story 1100 may include different AR characters talking to one another and/or talking to the user. In this case, the characters may generally perform the storyline plot through their actions and dialogue, and/or may talk directly to the user effectively including him/her as another character in the storyline. The interaction with the user may be for pure entertainment purposes or may provide an impetus for the user to perform an event transition trigger as described in the above example.
In some exemplary embodiments, a transition from one event to the next (see
For example, if the script of a story includes the following events (summarized to show transitions):
In the example, here 80% reading accuracy is required to transition between events. The degree of reading accuracy required to transition between events may be set to a default value or to different values for different events.
The AR story program 232 may animate or otherwise augment the display of text to be read and text being read. For example, if the text to be read is “Someone has been eating my porridge, and it's all gone!” (which is displayed on the screen of the device), then the individual words may be animated or highlighted or such as they are being read. In some cases, e.g., as an aid to reading, some form of animation (e.g., a pointer or bouncing ball or the like) may be used to show the user which words are to be read next.
While the above example has been given for a particular story, those of ordinary skill in the art will appreciate and understand, upon reading this description, that this story is only an example, and a story played by the AR story program 232 may correspond to any sequence of events. The AR items may correspond to characters or items in the story (e.g., a person, animal, house, tree, etc.) or to a real-world item.
With reference to the flow diagram in
As used herein, rendering an event means rendering or otherwise playing or displaying the information associated with an event. Thus, as shown in
Once the event information is rendered (as described above), the program transitions to the next event (at 1316) in accordance with the event transition information for the event.
Although described above primarily with respect to a single user, the AR story program 232 may be used simultaneously by multiple users. As shown in
As should also be appreciated, when multiple devices use the AR story program 232 for the same story (thereby sharing a unified story space), the story flow may be controlled by more than one of the devices. For example, a parent and child in the same location or in different physical or geographic locations may each use their respective device to provide a story. The child's device may provide the primary live real-world view (i.e., the real-world objects 1414) and the AR story program 232 may provide the virtual objects 1416. Each of the child and parent will have a view of the unified story space 1412, update in real time as the story progresses. Alternatively, the parent's device may provide the primary live real-world view.
In another example of the operational aspects of the AR program 232, the experience of the stories and storytelling described above may be embellished by including the functionalities of several additional mechanisms of AR App 232. For example, using his/her device, a user may experience the storytelling AR environment and storyline events provided by the AR App 232 as described in the Stories and Storytelling example in this specification. However, in addition to the mechanisms already engaged by the AR App 232 to perform the storytelling experience, the AR App 232 may also engage its facial recognition mechanism 236, its facial manipulation mechanism 238, its animation mechanism 242 and its augmenter mechanism 234 to modify AR elements of the story and/or to add additional AR elements to the AR story.
For example, the AR App 232 may augment one or more of the characters in the AR story (e.g., the Goldilocks character) with the user's facial expressions in real time.
Using this example, with reference again to
The AR App 232 may then engage its augmenter mechanism 234 to render or otherwise superimpose the manipulated image of the user's face onto a character of the story (e.g., Goldilocks). The augmenter 234 mechanism may also employ the animation mechanism 242 to animate the images prior to or while combining them with the AR images in the AR environment. The result may be a real time view of the character in the AR storyline augmented with an image of the user's face, including the user's real time facial expressions. For example, if the user U1 smiles, the face augmented onto the face of the character may also smile, if the user frowns, the face augmented onto the face of the character may also frown, and so on. As can be appreciated, any other type of facial expression may also be translated from the user to the AR character in real time by this process. Note that it may be preferable for the augmented face of the user, and the user's expressions, to be augmented onto the face of the AR character in such a way that the resulting face appears to be the natural face of the character.
As an extension of this example (or alone), the AR App 232 may also engage its gesture recognition mechanism 252 in order to capture the bodily gestures of a user, e.g., as captured by the front camera 1002 (
It can be appreciated that the other functionalities and flows of the storytelling program (AR App 232) as described in the storytelling example in other areas of this specification may also apply to this example. For instance, the event flow depicted in
Note that if multiple users are viewing the same story on their individual devices 1000 collectively or as collaboration (e.g., as shown in
Note also that a first user's facial expressions may be mapped onto a first AR character, and the first user's bodily expressions may be mapped onto a second AR character. Using the example above with multiple users on multiple devices, Goldilocks may perform the first user's facial expressions and the second user's bodily gestures, and Papa Bear may perform the second user's facial expressions and the first user's bodily gestures. It can be appreciated that any combination thereof, as well as any combination that may include additional AR characters and/or additional users may also be provided.
Similarly, each user's voice (possibly augmented, e.g., using speech or voice augmentation mechanism(s) 248) may be associated with and mapped onto a particular story character. Thus, e.g., a particular user (e.g., a father) may be associated with an AR character (such as the Papa Bear) such that the father's facial expressions and/or bodily gestures are mirrored by the AR character, and the father's voice is reproduced, possibly augmented, by the AR character. This approach may be more useful or effective when the users are not in the same location.
It should also be noted that this functionality may not necessarily require for the AR characters to be a part of a story as described in the Storytelling with AR App 232 example. For example, the users' may portray themselves as avatars or as other types of representations of themselves within the AR environment. If multiple users are experiencing the AR environment simultaneously, then each user may view the other users within the AR environment augmented with each user's respective facial expressions and/or bodily gestures. Also, as with the example above, the facial expressions of one user may be mapped onto a first representation while the bodily gestures of the same user may be mapped onto a second representation within the AR environment. It should be appreciated that any combination thereof may also be provided by AR App 232.
In this manner, users may effectively animate their corresponding representations (e.g., avatars) in the shared and unified virtual space.
In other exemplary functionality of embodiments of the AR program 232, a user's device may be associated with a virtual character or object in the virtual space. In such case, movement of the user's device may be used to animate the corresponding virtual character or object.
For example, a user may have a device (corresponding to device 200 in
In these exemplary embodiments, the AR App 232 may engage one or more of the device's sensors 228 (
In these exemplary embodiments, the AR App 232 may augment the AR environment by placing a virtual object corresponding to the device into the AR environment represented as a virtual object. The virtual object may be superimposed into the hand of an AR character (so that it may appear that the character is holding the object), or it may be free standing as a standalone virtual object (e.g., a first-person object, cursor, etc.), or it may be placed in combination with other virtual or other real-world objects, or any combination thereof.
In an example, the user may physically move their device with their hands (up and down, side to side, rotate it, or induce any other type of physical motion or movement onto the device), and the corresponding virtual object within the AR environment may follow a similar motion or movement within the view on the device.
Note that the device may be mapped into the AR environment as any type of virtual object, including a virtual character (or avatar) or onto any combination of virtual objects. In the case of a multi-user interaction, a user's device may thus be seen by the other users in the AR environment as a virtual object.
Note that the examples described above are only for demonstrational purposes and do not limit the scope of the current invention to only those examples listed. As such, it will be immediately appreciated by a person of ordinary skill in the art that the AR App 232 may represent the device as a virtual object of any type of form, including the form of the device itself.
In addition, the AR App 232 may employ the native controls of the device and use them to allow the user to operate the virtual object as it is displayed in the AR environment.
In addition, it may be possible for more than one user to participate in the AR environment of this example. By adding and/or combining other mechanisms of AR App 232 and functionalities described in other examples in this specification (e.g. the communications mechanism 240) additional participants with additional devices may also participate.
In other exemplary functionality of embodiments of the AR program 232, e.g., as shown in
In this example, both user U1 and user U2 may view the same or similar AR environment on each of their respective devices as shown in
Note however that device 1602 may also be considered a primary device, such that both devices 1600, 1602 may instead view the images captured by its rear camera 1608′. In addition, an extension of this example may be that both devices 1600, 1602 may be primary devices and the AR view may be a combination of augmented views captured by both rear cameras 1608, 1608′ from both devices 1600, 1602.
Although this example shows only two users/devices, it should be appreciated that multiple users/devices may participate in the viewing of the shared AR environment and any one or more of the user's devices may be primary devices.
In this example, the rear camera 1608 of U1's device 1600 may continuously capture images of U1's immediate environment and both users U1 and U2 may view the images in real time. Note that the images may or may not be augmented with virtual images or information as described in any of the other examples presented herein. The first user U1 may physically move from one location to another location within their real-world environment (e.g., the user U1 may walk around) and their device 1600 may continue to capture and save to memory the changing images of the view as he/she moves. As user U1 may continuously move within their environment while capturing and storing the real-world view, the AR App 232 may create and map a 2-D or, preferably, a 3-D model of user U1's real world environment (e.g. using 2-D and 3-D modeling mechanism 244) on his/her device 1600.
To accomplish this, the AR App 232 on U1's device 1600 may employ the device's various sensors 228 (such as an accelerometer and a gyroscope), as well as the device's GPS module 229. By recording the device's location (via the GPS module 229), the devices orientation (e.g., via the gyroscope) and the device's movement (e.g., via the accelerometer), and correlating this data with the real-time captured images of the environment, the AR App 232 may map the captured views with the location, orientation and movement information to create a 2-D or, preferably, a 3-D model of the environment as viewed by user U1. The 2-D and 3-D modeling mechanism 244 may include modeling algorithms and software necessary to create the model of the environment utilizing that various data captured by the device 1600 and/or the AR App 232. It can be seen that as user U1 may continue to move about within his/her environment while continuously capturing additional data, the 2-D or 3-D model of his/her environment may become more robust, comprehensive and filled with more details of the environment.
Note that virtual objects, characters or other forms may also be augmented into the environment and included in the model described above for each participant to view and experience on their respective devices 1600, 1602. The virtual forms may be static, dynamic, moving or any combination thereof. In this way, each user may experience a fully augmented AR environment from their own unique perspective.
As the model of the environment is created by AR App 232, the App 232 may continuously communicate model data to user U2's device 1602. In this way, user U2 may also view the modeled environment. In addition, because the model may include mapped 2-D or 3-D data, the user U2 may also move about the modeled environment by physically moving. That is, the user U2 may physically move in his/her own environment, and simultaneously, as viewed on his/her device 1602, may correspondingly move about in the mapped 2-D or 3-D model of the user U1's environment.
To accomplish this, the AR App 232 on the user U2's device 1602 may also engage its device's sensors 228 (e.g. an accelerometer, a gyroscope, etc.) and its device's GPS system 229. In this way, the AR App 232 may determine the physical location of the user U2 within his/her real-world environment, and may then map this location to a virtual location within the modeled 3-D environment. Then, when the user U2 may physically move within his/her real-world environment, the AR App 232 may calculate the exact direction, distance and other aspects of U2's movement relative to his/her prior location. Equipped with this data, the AR App 232 may correlate the movement with the 3-D model and map the movement within the modeled environment. The AR App 232 may then apply the data to the view as seen by user U2 on his/her device 1602 so that the resulting view may represent the movement within the AR environment.
For example, the user U2 may physically take a step forward in their real-world environment, and simultaneously experience a forward step in the AR environment as viewed on their device 1602. Expanding upon this example, consider that the real-world environment of the user U1 may include a house, and the house may thusly be viewed by both users U1 and U2 on their respective devices 1600, 1602 (
It should be noted that the user U2's experience may not rely on the real-time view of the user U1's camera(s), but instead may rely on the modeled data and the coordinates and movements of the user U2 as described above. In this way, using the example presented, the user U1 may be on one side of the house and the user U2 may be on an opposite side of the house and each user may view their respective (and different) views of the environment.
While the exemplary embodiments have been described with respect to a device such as a smartphone or a tablet computer or the like, those of ordinary skill in the art will appreciate and understand, upon reading this description, that different and/or other devices may be used. For example, in some embodiments, the cameras may not be in the same device. Furthermore, in some embodiments, the device may be an AR glasses or the like with one or more front-facing cameras.
In some other exemplary embodiments, the rear view may be obtained by direct viewing. For example, in embodiments in which the device is incorporated (fully or partially) into AR glasses, the user may view the scene (in front of them) without the use of (or need for) a rear-facing camera to capture the environment. In such embodiments, the user's eyes effectively act as a rear-facing camera (facing away from the user), and a rear-facing camera is not needed, although one may be used, e.g., to supplement or record the user's view. In such embodiments, the device 200 may exclude rear camera(s) 212.
In such embodiments, a hypothetical avatar may be animated over the live environment seen through the AR glasses lens. One or more front facing cameras may capture the user's facial expressions and map them onto the avatar. When the user is wearing the device (e.g., a VR headset or AR glasses), the user's expression may be determined, e.g., using one or more cameras looking at the user's eyes and/or mouth and/or facial muscle sensing.
The applications, services, mechanisms, operations, and acts shown and described above are implemented, at least in part, by software running on one or more computers.
Programs that implement such methods (as well as other types of data) may be stored and transmitted using a variety of media (e.g., computer readable media) in a number of manners. Hard-wired circuitry or custom hardware may be used in place of, or in combination with, some or all of the software instructions that can implement the processes of various embodiments. Thus, various combinations of hardware and software may be used instead of software only.
One of ordinary skill in the art will readily appreciate and understand, upon reading this description, that the various processes described herein may be implemented by, e.g., appropriately programmed general purpose computers, special purpose computers and computing devices. One or more such computers or computing devices may be referred to as a computer system.
According to the present example, the computer system 1700 includes a bus 1702 (i.e., interconnect), one or more processors 1704, a main memory 1706, read-only memory 1708, removable storage media 1710, mass storage 1712, and one or more communications ports 1714. Communication port(s) 1714 may be connected to one or more networks (not shown) by way of which the computer system 1700 may receive and/or transmit data.
As used herein, a “processor” means one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or like devices or any combination thereof, regardless of their architecture. An apparatus that performs a process can include, e.g., a processor and those devices such as input devices and output devices that are appropriate to perform the process.
Processor(s) 1704 can be any known processor, such as, but not limited to, an Intel® Itanium® or Itanium 2® processor(s), AMD® Opteron® or Athlon MP® processor(s), or Motorola® lines of processors, and the like. Communications port(s) 1714 can be any of an Ethernet port, a Gigabit port using copper or fiber, or a USB port, and the like. Communications port(s) 1714 may be chosen depending on a network such as a Local Area Network (LAN), a Wide Area Network (WAN), or any network to which the computer system 1700 connects. The computer system 1700 may be in communication with peripheral devices (e.g., display screen 1716, input device(s) 1718) via Input/Output (I/O) port 1720.
Main memory 1706 can be Random Access Memory (RAM), or any other dynamic storage device(s) commonly known in the art. Read-only memory (ROM) 1708 can be any static storage device(s) such as Programmable Read-Only Memory (PROM) chips for storing static information such as instructions for processor(s) 1704. Mass storage 1712 can be used to store information and instructions. For example, hard disk drives, an optical disc, an array of disks such as Redundant Array of Independent Disks (RAID), or any other mass storage devices may be used.
Bus 1702 communicatively couples processor(s) 1704 with the other memory, storage and communications blocks. Bus 1702 can be a PCI/PCI-X, SCSI, a Universal Serial Bus (USB) based system bus (or other) depending on the storage devices used, and the like. Removable storage media 1710 can be any kind of external storage, including hard-drives, floppy drives, USB drives, Compact Disc-Read Only Memory (CD-ROM), Compact Disc-Re-Writable (CD-RW), Digital Versatile Disk-Read Only Memory (DVD-ROM), etc.
Embodiments herein may be provided as one or more computer program products, which may include a machine-readable medium having stored thereon instructions, which may be used to program a computer (or other electronic devices) to perform a process. As used herein, the term “machine-readable medium” refers to any medium, a plurality of the same, or a combination of different media, which participate in providing data (e.g., instructions, data structures) which may be read by a computer, a processor or a like device. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random-access memory, which typically constitutes the main memory of the computer. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves and electromagnetic emissions, such as those generated during radio frequency (RF) and infrared (IR) data communications.
The machine-readable medium may include, but is not limited to, floppy diskettes, optical discs, CD-ROMs, magneto-optical disks, ROMs, RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing electronic instructions. Moreover, embodiments herein may also be downloaded as a computer program product, wherein the program may be transferred from a remote computer to a requesting computer by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., modem or network connection).
Various forms of computer readable media may be involved in carrying data (e.g. sequences of instructions) to a processor. For example, data may be (i) delivered from RAM to a processor; (ii) carried over a wireless transmission medium; (iii) formatted and/or transmitted according to numerous formats, standards or protocols; and/or (iv) encrypted in any of a variety of ways well known in the art.
A computer-readable medium can store (in any appropriate format) those program elements which are appropriate to perform the methods.
As shown, main memory 1706 is encoded with application(s) 1722 that support(s) the functionality as discussed herein (the application(s) 1722 may be an application(s) that provides some or all of the functionality of the services/mechanisms described herein, e.g., AR story application 232,
During operation of one embodiment, processor(s) 1704 accesses main memory 1706 via the use of bus 1702 in order to launch, run, execute, interpret or otherwise perform the logic instructions of the application(s) 1722. Execution of application(s) 1722 produces processing functionality of the service related to the application(s). In other words, the process(es) 1724 represent one or more portions of the application(s) 1722 performing within or upon the processor(s) 1704 in the computer system 1700.
For example, process(es) 1704 may include an AR application process corresponding to AR application 232.
It should be noted that, in addition to the process(es) 1724 that carries(carry) out operations as discussed herein, other embodiments herein include the application 1722 itself (i.e., the un-executed or non-performing logic instructions and/or data). The application 1722 may be stored on a computer readable medium (e.g., a repository) such as a disk or in an optical medium. According to other embodiments, the application 1722 can also be stored in a memory type system such as in firmware, read only memory (ROM), or, as in this example, as executable code within the main memory 1706 (e.g., within Random Access Memory or RAM). For example, application(s) 1722 may also be stored in removable storage media 1710, read-only memory 1708, and/or mass storage device 1712.
Those skilled in the art will understand that the computer system 1700 can include other processes and/or software and hardware components, such as an operating system that controls allocation and use of hardware resources. For example, as shown in
As discussed herein, embodiments of the present invention include various steps or operations. A variety of these steps may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a general-purpose or special-purpose processor programmed with the instructions to perform the operations. Alternatively, the steps may be performed by a combination of hardware, software, and/or firmware. The term “module” refers to a self-contained functional component, which can include hardware, software, firmware or any combination thereof.
One of ordinary skill in the art will readily appreciate and understand, upon reading this description, that embodiments of an apparatus may include a computer/computing device operable to perform some (but not necessarily all) of the described process.
Embodiments of a computer-readable medium storing a program or data structure include a computer-readable medium storing a program that, when executed, can cause a processor to perform some (but not necessarily all) of the described process.
Where a process is described herein, those of ordinary skill in the art will appreciate that the process may operate without any user intervention. In another embodiment, the process includes some human intervention (e.g., a step is performed by or with the assistance of a human).
Although embodiments hereof are described using an integrated device (e.g., a smartphone), those of ordinary skill in the art will appreciate and understand, upon reading this description, that the approaches described herein may be used on any computing device that includes a display and at least one camera that can capture a real-time video image of a user. For example, the system may be integrated into a heads-up display of a car or the like. In such cases, the rear camera may be omitted.
As used herein, including in the claims, the phrase “at least some” means “one or more,” and includes the case of only one. Thus, e.g., the phrase “at least some ABCs” means “one or more ABCs”, and includes the case of only one ABC.
The term “at least one” should be understood as meaning “one or more”, and therefore includes both embodiments that include one or multiple components. Furthermore, dependent claims that refer to independent claims that describe features with “at least one” have the same meaning, both when the feature is referred to as “the” and “the at least one”.
As used in this description, the term “portion” means some or all. So, for example, “A portion of X” may include some of “X” or all of “X”. In the context of a conversation, the term “portion” means some or all of the conversation.
As used herein, including in the claims, the phrase “based on” means “based in part on” or “based, at least in part, on,” and is not exclusive. Thus, e.g., the phrase “based on factor X” means “based in part on factor X” or “based, at least in part, on factor X.” Unless specifically stated by use of the word “only”, the phrase “based on X” does not mean “based only on X.”
As used herein, including in the claims, the phrase “using” means “using at least,” and is not exclusive. Thus, e.g., the phrase “using X” means “using at least X.” Unless specifically stated by use of the word “only”, the phrase “using X” does not mean “using only X.”
As used herein, including in the claims, the phrase “corresponds to” means “corresponds in part to” or “corresponds, at least in part, to,” and is not exclusive. Thus, e.g., the phrase “corresponds to factor X” means “corresponds in part to factor X” or “corresponds, at least in part, to factor X.” Unless specifically stated by use of the word “only,” the phrase “corresponds to X” does not mean “corresponds only to X.”
In general, as used herein, including in the claims, unless the word “only” is specifically used in a phrase, it should not be read into that phrase.
As used herein, including in the claims, the phrase “distinct” means “at least partially distinct.” Unless specifically stated, distinct does not mean fully distinct. Thus, e.g., the phrase, “X is distinct from Y” means that “X is at least partially distinct from Y,” and does not mean that “X is fully distinct from Y.” Thus, as used herein, including in the claims, the phrase “X is distinct from Y” means that X differs from Y in at least some way.
It should be appreciated that the words “first” and “second” in the description and claims are used to distinguish or identify, and not to show a serial or numerical limitation. Similarly, the use of letter or numerical labels (such as “(a)”, “(b)”, and the like) are used to help distinguish and/or identify, and not to show any serial or numerical limitation or ordering.
No ordering is implied by any of the labeled boxes in any of the flow diagrams unless specifically shown and stated. When disconnected boxes are shown in a diagram the activities associated with those boxes may be performed in any order, including fully or partially in parallel.
As used herein, including in the claims, singular forms of terms are to be construed as also including the plural form and vice versa, unless the context indicates otherwise. Thus, it should be noted that as used herein, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.
Throughout the description and claims, the terms “comprise”, “including”, “having”, and “contain” and their variations should be understood as meaning “including but not limited to”, and are not intended to exclude other components.
The present invention also covers the exact terms, features, values and ranges etc. in case these terms, features, values and ranges etc. are used in conjunction with terms such as about, around, generally, substantially, essentially, at least etc. (i.e., “about 3” shall also cover exactly 3 or “substantially constant” shall also cover exactly constant).
Use of exemplary language, such as “for instance”, “such as”, “for example” and the like, is merely intended to better illustrate the invention and does not indicate a limitation on the scope of the invention unless so claimed. Any steps described in the specification may be performed in any order or simultaneously, unless the context clearly indicates otherwise.
All of the features and/or steps disclosed in the specification can be combined in any combination, except for combinations where at least some of the features and/or steps are mutually exclusive. In particular, preferred features of the invention are applicable to all aspects of the invention and may be used in any combination.
Reference numerals have just been referred to for reasons of quicker understanding and are not intended to limit the scope of the present invention in any manner.
Thus, is provided an augmented reality system that combines a live view of a real-world, physical environment with imagery based on live images from one or more other devices.
While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiment, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
This application is a continuation of PCT/IB2018/052882, filed Apr. 26, 2018, the entire contents of which are hereby fully incorporated herein by reference for all purposes. PCT/IB2018/052882 claims priority from U.S. Provisional Applications (i) No. 62/503,868, filed May 9, 2017, (ii) No. 62/503,826, filed May 9, 2017, (iii) No. 62/513,208, filed May 31, 2017, (iv) No. 62/515,419, filed Jun. 5, 2017, and (v) No. 62/618,388, filed Jan. 17, 2018, the entire contents of all of which is hereby fully incorporated herein by reference for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62503826 | May 2017 | US | |
| 62503868 | May 2017 | US | |
| 62513208 | May 2017 | US | |
| 62515419 | Jun 2017 | US | |
| 62618388 | Jan 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/IB2018/052882 | Apr 2018 | US |
| Child | 16675196 | US |
