Patent Application
20180025544
Devices for implementing and interacting with virtual content in an augmented reality become more and more common. Augmented reality (AR) relates to a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data.
Examples of virtual objects in AR may be graphical objects that are positioned at predefined positions in the real world, for example graphical notes overlaid to a machine for supporting a technician in operating and/or servicing the machine. For example, a user may wear AR glasses comprising a video camera for capturing substantially the same field of vision that the user sees. The AR glasses further comprise a display or projector for rendering computer-generated graphics within the glasses so that the user experiences an augmented reality.
There are many other applications that use augmented reality comprising: industry, medicine, travel industry, gaming industry, advertising industry, science, military, navigation, office workplaces, sport and entertainment, stock markets, translation, visual art, and much more. Even though the present invention mainly refers to AR glasses for implementing the invention, other AR devices are available such as smart phones, smart contact lenses or head-up displays in cars or airplanes etc.
Problems in the prior art arise when rendering parameters for the virtual content to be displayed is to be determined. For example, when essential information has to be displayed at a certain point in the augmented reality, it is not always easy to implement an accurate positioning of the virtual content relative to the real world. In case a user moves, the correct positioning is even more complex and requires high computing power to update rendering parameters, for example when a viewing angle or distance changes.
It is therefore the invention to provide an improved computer-implemented method and device for determining rendering information for virtual content within an augmented reality.
This object is solved by the subject matter of the independent claims.
Preferred embodiments are defined by the dependent claims.
According to an embodiment of the invention, a computer-implemented method for determining rendering information for virtual content within an augmented reality is provided. The method may comprise capturing an image comprising a graphical tag. For capturing the image, an image capturing unit may be used. The graphical tag may comprise one or more geometric objects, and the graphical tag may represent coded information.
The method may further comprise obtaining from the captured image size reference information. The size reference information may for example indicate a physical size of the graphical tag or other information that can be used to determine a distance between the graphical tag and the image capturing unit. According to an embodiment, the size reference information may be the physical size coded into the graphical tag, or may be a reference to an entry in a table comprising size reference information. In an embodiment, the size reference information may further be determined by presence of a second graphical tag that is located at a predetermined distance from the first graphical tag.
The method further comprises determining a distortion of a captured view of at least one of said geometric objects. For example, when the image capturing unit captures the image comprising the graphical tag from a non-perpendicular direction from the graphical tag, it may be important to determine the exact viewing angle and direction. In this regard, according to an embodiment of the invention, the changed aspect ratio of one or more geometric objects is determined due to the distortion of the viewing angle.
Further, the method comprises determining, based on the size reference information and the distortion of the captured view, a relative position of the graphical tag to the image capturing unit. For example, when the distance of the image capturing unit and the graphical tag is determined, e.g. based on the size reference information, and the viewing angle and viewing direction between the graphical tag and the image capturing unit is determined, e.g. based on the determined distortion, the relative position of the graphical tag to the image capturing unit may be determined in a very precise manner.
As a next step, the method comprises determining, based on the determined relative position, positioning information and scaling information for rendering the virtual content within an augmented reality relative to the graphical tag. Since the relative positioning (i.e. a distance, a direction and an angle) between the graphical tag and the image capturing unit are known, the positioning information and scaling information, i.e. the required rendering information are readily determined.
Based on the determined rendering information, the virtual content may be rendered within the augmented reality using the determined positioning information and scaling information. For example, a virtual graphical object may be displayed in AR glasses of a user at a predefined position that can be easily identified due to the present invention.
According to an embodiment of the present invention, the coded information may comprise a source for the virtual content and/or the size reference information. For example, the graphical tag may be a scannable code that references to a web address where the virtual content may be downloaded. In addition to the virtual content itself, the size reference information may be downloaded from the web address or the size reference information may be coded within the graphical tag.
According to an embodiment of the present invention, determining the relative position of the graphical tag to the image capturing unit may comprise determining a distance, a direction and an angle between the graphical tag and the image capturing unit. Determining the relative position of the graphical tag to the image capturing device may further comprise determining a degree of distortion and a direction of distortion of the one or more geometric objects, wherein the degree of distortion is determined by analyzing an aspect ratio of at least one of said one or more geometric objects.
According to an embodiment of the present invention, the determining the positioning information and scaling information may comprise dynamically updating the orientation of the virtual content by updating the positioning information and scaling information when the relative position of the graphical tag and the image capturing unit changes. For example, a change of the distortion or the distance may be monitored and based on the monitoring, a movement of the image capturing device or the graphical tag may be determined. Thus, the method can quickly react to such positioning changes and update the rendering parameters.
According to an embodiment of the present invention, the coded information may further comprise an indication of a category of the virtual content. For example, the indication of the category may relate to digital rights management (DRM) or to a specification of the content itself, such as advertising, shopping information, age-based restrictions, and so on.
An embodiment of the invention relates to a corresponding device for determining rendering information for virtual content within an augmented reality. The device may be implemented at least as part of, for example, a head up display, smart glasses, a mobile phone, a tablet PC, etc. However, the embodiments of the present invention are not limited to one of the specific hardware configurations, but may be implemented in any hardware that facilitates an environment for implementing the described methods of the invention.
The above and other aspects, features, and advantages of the present invention will be more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:
Embodiments of the present invention provide a concept of improving management of rendering parameters for rendering virtual objects in an augmented reality. Additionally, an embodiment of the present invention relates to the obtaining of virtual content that is to be rendered as part of the augmented reality.
The data communications interface of the AR glasses 100 may be, for example, a Bluetooth interface, a Wi-Fi interface, or any other wireless interface for exchanging data with a network or with another device, such as a smart phone. For example, data may be exchanged with the Internet by using a wireless interface of the AR glasses 100 that connects the AR glasses 100 with a smart phone that is connected to the Internet. However, the AR glasses 100 may also comprise an interface for directly communicating with the Internet, such as by using Wi-Fi and/or an interface of the cellular standard, such as LTE or other GSM/EDGE and UMTS/HSPA network technologies.
According to an embodiment, the user may not necessarily wear AR glasses 100 but may instead use another AR device, such as a smart phone or any other device for augmenting reality.
Referring to
The graphical tag 102 may comprise a scannable code that references to and/or comprises coded information. As will be discussed in greater details below, the coded information of the graphical tag 102 may comprise different information items, such as size reference information. The encoded size reference information of the graphical tag 102 may be the actual physical size of the scannable code, i.e. of the graphical tag, that may be required for determining a distance between the graphical tag 102 and the image capturing unit. The graphical tag 102 may be, for example, a modified quick response code (QR code), or any other matrix barcode. More details of an exemplary graphical tag will be described with reference to
According to
The coded information of the graphical tag 102 may comprise information for obtaining virtual content 204 to be displayed as part of the augmented reality. The coded information may for example comprise an URL to the virtual content so that the virtual content 204 can be downloaded from that URL and displayed via the AR glasses 100 at a predefined position. Thus, the coded information may further comprise instructions for determining positioning information of how to render the virtual content 204 relative to the graphical tag 102.
The virtual content 204 can be of any form and kind that may be displayed or rendered as part of an augmented reality. In the example of
In the following, some non-limiting examples of applications for the present invention are provided. In a first example, the graphical tag 204 may be mounted on a wall of a house.
When a user wearing the AR glasses 100 comes close enough so that the image capturing unit captures an image of the graphical tag 102, the coded information of the graphical tag 102 is decoded and a relative position of the AR glasses 100 to the graphical tag 102 is determined. The determining of the relative position will be discussed in more detail below. The coded information of the graphical tag may, for example, be a reference to a 3D graphic that is to be rendered within the augmented reality, such as virtual decoration of the house. As such, in the above example, the user wearing the AR glasses 100 may see additional decoration of the house, such as a different painting of the walls, plants and trees or more abstract virtual content, such as the 3D image of a skyscraper instead of the house.
In another example, the graphical tag 102 may be used for interactive computer games, where one or more graphical tags 102 may be mounted at specific locations and users wearing AR glasses 100 can interact with virtual content that is displayed in the area around the graphical tag(s) 102. By using the technique described by the present invention, users may experience a much more realistic behavior of virtual objects as the virtual content according to the present invention can be placed at a defined position more accurately without being effectuated by unrealistic movement of the virtual content due to a movement of the user. In other words, many computer games already exist where smart phones or smart glasses are used for augmenting the reality and presenting virtual objects to the user that are part of the computer game. A user can then interact with the virtual objects (virtual content) through the AR device. However, when the AR device moves, the rendering parameters of the virtual object are most often not updated and adapted in a sufficient and realistic manner. This problem may be solved by the present invention, as will be discussed in greater detail below.
In still another example, the graphical tag may be positioned at locations where a user input through a keyboard may be required. For example, due to the very stable positioning of virtual content 204 within the augmented reality, the virtual content 204 may be a virtual keyboard for receiving user inputs. Such an embodiment may be advantageous in food industry, for example in aseptic environments, where operating with machines has to be sterile. As such, servicing or operating with sterile machines in aseptic environments may no longer require a physical keyboard or physical buttons, as virtual keyboards and buttons may be placed as augmented reality at predefined positions and locations. Thus, users or operators of the machines do not have to touch anything in critical environments, but can still make user inputs. Due to the very stable positioning of virtual content in the augmented reality, it may even be possible to interact with rather small keys on a keyboard. In particular, since embodiments of the present invention allow positioning of virtual content in an augmented reality in such a fixed and precise manner, a user may even interact with small virtual objects, such as virtual keys of a virtual keyboard. Thus, instead of installing a physical keyboard in the real world, a graphical tag 102 referencing a virtual keyboard may be mounted.
As described in the above example, embodiments of the present invention do not only provide passive virtual content, such as decoration objects or 3D graphics, but may further provide interactive virtual content that can be manipulated through user inputs and user interactions. For example, the image capturing unit of the AR glasses 100 may further capture the hand and/or fingers of the user and determine a user interaction with the displayed virtual content. When such a user interaction is determined, the virtual content may be altered or changed and so on.
There are many further examples of applications for the present invention. For example, the graphical tag 102 may be a wearable graphical tag, such that it is printed on a shirt. When the shirt is then viewed through AR glasses 100, the shirt may appear in a different design or color. In still another embodiment, the graphical tag 102 may reference to multimedia applications, where virtual content 204 is multimedia content that may be downloaded and displayed to the user, such as animated advertising or videos. Since some AR devices may further comprise a sound output unit, such as loudspeaker, the virtual content 204 may not only be a graphical content, but may further comprise sound.
Since users wearing the AR glasses 100 may not be willing to receive all available virtual content, the coded information may further comprise an indication of a category of the virtual content. Additionally, the indication of the category may relate to digital rights management (DRM) or to a specification of the content itself, such as advertising, shopping information, age-based restrictions, and so on. For example, the AR glasses 100 may be set to ignore virtual content 204 of the category “advertising”. In one embodiment, the user may only be allowed to download and render the virtual content 204 if the user has the corresponding rights (e.g. DRM, age verification, privacy settings, etc.).
As mentioned above, the coded information may not only refer to the virtual content 204, but may further comprise size reference information. For example, the size reference information may indicate an actual size of the graphical tag 102 or other information that can be used to determine a distance between the graphical tag and the image capturing unit. According to an embodiment, the size reference information may be the physical size coded into the graphical tag, or may be a reference to an entry in a table comprising size reference information. In an embodiment, the size reference information may further be determined by presence of a second graphical tag that is located at a predetermined distance from the first graphical tag. An example for using two graphical tags will be explained with regard to
The geometric objects 312 and 314 may be used to determine a viewing direction of the image capturing unit towards the graphical tag 102. In particular, by determining a distortion of the captured view of one or more of the geometric objects 312 and 314, the viewing direction may be determined. Thus, based on the size reference information and the distortion of the captured view, the relative position of the graphical tag 102 to the image capturing unit may be determined, as will be discussed in greater detail with regard to
To further increase accuracy of the relative position of the graphical tag to the image capturing unit, an angle around the connecting line between image capturing unit and graphical tag 102 may further be determined, as will be discussed in greater detail with regard to
As can be seen in
As a consequence, the relative position of the graphical tag 102 to the image capturing unit may be determined based on the size reference information and the determined distortion of the captured view of at least one of the geometric objects.
It should be understood that the square 314 of
It should be understood that the example of the square 314 and the dot 312 is for illustrative purposes only and should not be understood as a limitation of the present invention. Other geometric aspects may also be used, such as the shape of an arrow, or a rectangle, or more complex geometric objects that allow the determination of an orientation angle.
After the relative position of the graphical tag to the image capturing device has been determined, the rendering information may be determined. The rendering information may comprise positioning information of the virtual content 204 relative to the graphical tag and scaling information for rendering the virtual content 204 within the augmented reality.
The determining of the positioning information and scaling information for rendering the virtual content 204 within the augmented reality may further be based on rendering instructions contained in the coded information of the graphical tag 102. The rendering instructions may specify that the virtual object 204 is to be displayed at a predetermined relative distance and direction from the graphical tag 102. For example, the rendering instructions may specify that the virtual object 204 is to be displayed 1 m above and 2 m on the left side of the graphical tag 102.
As such, when the rendering information has been successfully determined, the virtual content 204 may be rendered, i.e. displayed or projected, within the augmented reality using the determined positioning information and scaling information.
The above described technique for determining the rendering information, i.e. the positioning information and the scaling information for rendering the virtual content, may be dynamically repeated for updating the rendering information continuously. For example, when the user moves, the rendering information is automatically updated so that the user experiences a very realistic view of the virtual content and the augmented reality.
This may be advantageous for larger virtual content.
According to an embodiment, the distance L1 between the two graphical tags 602a and 602b may be part of the coded information of the graphical tags 602a and 602b. In other words, the graphical tag 602a comprises coded information of the distance and direction towards the graphical tag 602b, and vice versa. Thus, when the distance L1 is known, the distance L2 between the graphical tags 602a/602b and the image capturing unit 600 in
