The present specification relates to rendering a message within a volumetric space.
Virtual reality (VR) is a rapidly developing area of technology in which video content is provided to a VR display system. As is known, a VR display system may be provided with a live or stored feed from a video content source, the feed representing a VR space or world for immersive output through the display system. In some embodiments, audio is provided, which may be spatial audio. A VR headset may be configured to provide VR video and audio content to the user, e.g. through the use of a pair of video screens and headphones incorporated within the headset.
Mixed reality (MR) is an area of technology in which real and virtual worlds are combined such that physical and digital objects co-exist and interact in real time.
Augmented reality (AR) refers to a real-world view that is augmented by computer-generated sensory input. In the context of the present specification, the term mixed reality is intended to encompass augmented reality.
Immersive content may be provided, for example, in virtual reality, augmented reality or mixed reality scenes. There remains a need for controlling the display of objects within such scenes.
In a first aspect, this specification describes an apparatus comprising: means for determining a position of a plurality of users, each having a different position within a volumetric space (which space may be one or more of a virtual reality, augmented reality or mixed reality space); and means for determining a rendering of a message within the volumetric space based, at least in part, on the determined positions of said users, such that the message is one or more of: visible to, audible by and/or touchable by each of said users. The apparatus may further comprise means for rendering the message based on the determined rendering.
In some embodiments, there is provided means for determining an orientation of each of said users. The said means for determining the orientation of each of said users may include means for determining a gaze direction of each of said users. The means for determining the rendering of the message within the volumetric space may determine the rendering based, at least in part, on the determined orientation of said users.
The apparatus may further comprise means for determining a change in a position and/or orientation of one or more of said users, wherein said means for determining the rendering of the message includes updating the rendering of the message within the volumetric space based on the determined change.
In some embodiments, the means for determining the rendering of the message may comprise rendering the message within the volumetric space such that the message is available to all of said users that have not acknowledged the message.
In embodiments wherein the message is audible by the users, the means for determining the rendering of a message within the volumetric space may comprise: defining an audio volume of the message depending on the location of the users; and/or defining an audio directionality of the message depending on the location of the users.
The plurality of users may be relevant users, such that the message is directed to said relevant users.
The message may be one or more of an audio, visual image, visual video, text or tactile message.
In some embodiments, the said means may comprise: at least one processor; and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus.
In a second aspect, this specification describes a method comprising: determining a position of a plurality of users within a volumetric space; and determining a rendering of a message within the volumetric space based, at least in part, on the determined positions of said users, such that the message is one or more of: visible to, audible by and touchable by each of said users. The method may comprise rendering the message accordingly.
The method may further comprise determining an orientation of each of said users (e.g. by determining a gaze direction of each of said users). Determining the rendering of the message within the volumetric space may determine the rendering based, at least in part, on the determined orientation of said users.
The method may further comprise determining a change in a position and/or orientation of one or more of said users. The said means for determining the rendering of the message may include updating the rendering of the message within the volumetric space based on the determined change.
Rendering of a message within the volumetric space may further comprise rendering the message such that the message is available to all of said users that have not acknowledged the message.
In some forms of the invention, the message may be audible by the users. The means for determining a rendering of a message within the volumetric space may comprise: defining an audio volume of the message depending on the location of the users; and/or defining an audio directionality of the message depending on the location of the users.
In some embodiments, the volumetric space for at least some of the users may have different dimensions, wherein: the means for determining the rendering of the message may comprise rendering the message such that the message is available to all users regardless of the different volumetric space dimensions.
In a third aspect, this specification describes an apparatus configured to perform the method as described with reference to the second aspect.
In a fourth aspect, this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform any method as described with reference to the second aspect.
In a fifth aspect, this specification describes a computer program comprising instructions stored thereon for performing at least the following: determining a position of a plurality of users within a volumetric space; and determining a rendering of a message within the volumetric space based, at least in part, on the determined positions of said users, such that the message is one or more of: visible to, audible by and touchable by each of said users.
In a sixth aspect, this specification describes a non-transitory computer-readable medium comprising program instructions stored thereon for performing at least the following: determining a position of a plurality of users within a volumetric space; and determining a rendering of a message within the volumetric space based, at least in part, on the determined positions of said users, such that the message is one or more of: visible to, audible by and touchable by each of said users.
In a seventh aspect, this specification describes an apparatus comprising: at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to: determine a position of a plurality of users within a volumetric space; and determine a rendering of a message within the volumetric space based, at least in part, on the determined positions of said users, such that the message is one or more of: visible to, audible by and touchable by each of said users.
Example embodiments will now be described, by way of non-limiting examples, with reference to the following schematic drawings, in which:
Assume that a message is sent to a group of users including the first, second and third users 2 to 6. The message may, for example, be an advertisement, a communication to the group from inside or outside a virtual reality world, a status message related to the volumetric VR world etc. The message may be displayed within the volumetric space (see the example message 10, for example). However, if users are looking in different directions, the message may not be visible to all users.
The headset 20 receives the virtual reality or augmented reality content data, for example from a media player (not shown). Here, the media player may comprise a mobile phone, smartphone or tablet computer configured to play content through its display. For example, the media player may be a touchscreen device having a large display over a major surface of the device, through which video content can be displayed. The media player may be inserted into a holder of a headset 20. With such headsets 20, a smart phone or tablet computer may display visual data which is provided to a user's eyes via respective lenses in the headset 20.
The headset 20 may include means for determining the spatial position of the user and/or orientation of the user's head. This may be by means of determining the spatial position and/or orientation of the headset 20. Over successive time frames, a measure of movement may therefore be calculated and stored. For example, the headset 20 may incorporate motion tracking sensors which may include one or more of gyroscopes, accelerometers and structured light systems. These sensors may generate position data from which a current visual field-of-view (FOV) is determined and updated as the user, and so the headset 20, changes position and/or orientation. The headset 20 may comprise two digital screens for displaying stereoscopic video images of the virtual world in front of respective eyes of the user, and also two speakers for delivering audio, if provided. The example embodiments herein are not limited to a particular type of headset 20.
In some example embodiments, the spatial position and/or orientation of the user's head may be determined using a six-degrees-of-freedom (6DoF) method. As shown in
The display system 20 may be configured to display virtual reality or augmented reality content data to the users 2, 4 and 6 based on spatial position and/or the orientation of the headset 20 worn by the users. A detected change in spatial position and/or orientation, i.e. a form of movement, may result in a corresponding change in the visual data to reflect a position or orientation transformation of the users with reference to the volumetric space 1 into which the visual data is projected. This allows virtual reality content data to be consumed with the users experiencing a 3D virtual reality or augmented reality environment.
As described above, a message may be sent to the users within the volumetric space 1 and that message may be displayed to the first user 2, the second user 4 and the third user 6 within the volumetric space. If those users are looking in different directions, then it would be possible to overlay the volumetric content provided to each user with the message. However, overlaying messages on screens with potentially small fields of view and potentially blocking some other content may not provide a good user experience. Moreover, if the users are experiencing the same volumetric content, then it may be more natural if the presented message was the same for all users. For example, although users (such as the users 2 to 6) may be located in different parts of a volumetric space, a message may be rendered in the same part of the volumetric space 1 for all users, thereby creating an aligned perception of the volumetric space for each user.
The algorithm 30 includes an operation 34 in which orientations of the users are determined (for example, based on gaze directions of the users). One method for determine gaze direction is to use video-based eye-trackers. A camera may focus on one or both eyes of the user and record eye movement as the user looks at some visual stimulus. Eye-trackers may use the centre of the pupil and infrared/near infra-red non-collimated light to create corneal reflections (CR). The vector between the pupil centre and the corneal reflections can be used to compute the point of regard on a surface or the gaze direction. Other methods for detecting gaze direction include eye-attached tracking, movement of the eye being detected with a special contact lens with an embedded mirror or magnetic field sensor. One method is to use electric potentials measured with electrodes placed around the eye. It should be noted that determining the orientation of users is not required in all embodiments. For example, as described below, a message provided to users may be an audio message; in such circumstances, the orientation of the users may not be important.
At operation 36, appropriate rendering of the message (based, at least in part, on the position and/or orientation of the users) is determined. The messages are then rendered accordingly in operation 38.
As shown in
In operation 36, appropriate rendering of the message (based on the positions and/or orientations of the users) is determined. As shown on
In this way, the algorithm 30 can be used to modify the images of the message so that they may be optically perceivable from the locations of each of the users within the space 40. For example, the positions and/or orientations of the presented messages may be selected such that maximally large portions of the message are visible to the users. It should be noted that, in some embodiments, the users to whom the message is directed are a subset of the all users consuming the volumetric content within the space 40. In such circumstances, the algorithm 30 may modify the images of the message so that the message is optically perceivable from the locations of each of the users to whom the message is directed.
The determination could be made in other ways, such as using a camera to detect the locations of one or more users or detecting the locations of mobile communication devices (e.g. handsets) of one or more users.
The algorithm 50 includes an operation 54 in which acknowledgements of a message are determined. As described further below, a particular message may no longer be presented to a user once that user has acknowledged the message. A user may, for example, acknowledge a message by touching that message within the volumetric space. Acknowledgement of a message could be conducted in other ways, such as using gaze tracking or a voice input.
At operation 56, the rendering of the message is updated, based on changes in position and/or orientation of users and/or based on acknowledgements of a message.
The operation 52 of the algorithm 50 may determine changes in the positions of one or more of the first, second and third users 42′ to 46′ within the space 40′. Thus, for example, the first user 42′ may have moved to the right and rotated in an anti-clockwise direction (relative to the position and orientation of the first user 42 described above with reference to
The operation 54 of the algorithm 50 may determine that the third user 46′ has acknowledged the message 48c described above. This is represented in
As shown in
In operation 56, appropriate rendering of the message (based, at least in part, on the position and/or orientation of the users) is determined. As shown on
The field-of-view (e.g. gaze direction) of the third user 46′ is not shown in
As described above, the algorithm 50 may be used to modify the images of the message so that they may be optically perceivable from the locations of each of the users as those locations change.
As described above, the message presented to users may be a visual message. This is not essential to all embodiments. For example, the message may be one or more of audio, visual image, visual video, text and/or tactile message.
The audio message 78 may be presented based, at least in part, on the position and/or orientation of the users 72 to 76. For example, the volume of the audio message may be such that it is audible to the users to which it is directed. Moreover, the audio message may be directional (regardless of whether the original audio message included a directional component). For example, as shown in
As described above, the rendering of a message (such as the audio message 78) may be changed in response to an acknowledgement of the message by one or more of the users to whom the message is directed. For example, in the event that the third user 76 acknowledges the message 78, the audio message in the second direction 82 may be altered so that the audio is directed towards the second user 74 but not the third user 76.
As shown in
As will be apparent from the discussion above, whilst the algorithms 30 and 50 may determine the position of multiple users within a space (such as the volumetric spaces 40, 40′, 70 and 90) and render a message to multiple such users, the message may not necessarily be presented to all users within the space. For example, a message may be presented to a subset of users within the space (e.g. relevant users). Some users may be excluded, for example, because the message is not relevant to those users; other users may be excluded because they have acknowledged receipt of the message. In some embodiments, the message may be available (e.g. visible or audible) to users to whom the message is not directed; in other embodiments, the message may only be available (e.g. visible or audible) to users to whom the message is directed. In some embodiments, the position and/or orientation of all users within a volumetric space may be determined, but this is not essential to all example embodiments.
For completeness,
The processor 302 is connected to each of the other components in order to control operation thereof.
The memory 304 may comprise a non-volatile memory, a hard disk drive (HDD) or a solid state drive (SSD). The ROM 312 of the memory 304 stores, amongst other things, an operating system 315 and may store software applications 316. The RAM 314 of the memory 304 is used by the processor 302 for the temporary storage of data. The operating system 315 may contain code which, when executed by the processor, implements aspects of the algorithms 30 or 50.
The processor 302 may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors.
The processing system 300 may be a standalone computer, a server, a console, or a network thereof.
In some embodiments, the processing system 300 may also be associated with external software applications. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications may be termed cloud-hosted applications. The processing system 300 may be in communication with the remote server device in order to utilize the software application stored there.
Embodiments of the present invention may be implemented in software, hardware, application logic or a combination of software, hardware and application logic. The software, application logic and/or hardware may reside on memory, or any computer media. In an example embodiment, the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media. In the context of this document, a “memory” or “computer-readable medium” may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
Reference to, where relevant, “computer-readable storage medium”, “computer program product”, “tangibly embodied computer program” etc., or a “processor” or “processing circuitry” etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialised circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices and other devices. References to computer program, instructions, code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device as instructions for a processor or configured or configuration settings for a fixed function device, gate array, programmable logic device, etc.
As used in this application, the term “circuitry” refers to all of the following: (a) hardware-only circuit implementations (such as implementations in only analogue and/or digital circuitry) and (b) to combinations of circuits and software (and/or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a server, to perform various functions) and (c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.
If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined. Similarly, it will also be appreciated that the flow diagrams of
It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.
Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.
Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.
It is also noted herein that while the above describes various examples, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
18166947.4 | Apr 2018 | EP | regional |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/IB2019/052671 | 4/1/2019 | WO | 00 |