Patent Application
20150348322
1. Field of the Invention
The present invention relates in general to the field of information handling system display presentation, and more particularly to a dynamically composited information handling system augmented reality at a primary display.
2. Description of the Related Art
As the value and use of information continues to increase, individuals and businesses seek additional ways to process and store information. One option available to users is information handling systems. An information handling system generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, information handling systems may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in information handling systems allow for information handling systems to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications. In addition, information handling systems may include a variety of hardware and software components that may be configured to process, store, and communicate information and may include one or more computer systems, data storage systems, and networking systems.
Information handling systems are commonly used to play games, often including interactive games in which an end user competes with other players through an Internet interface. The type and complexity of games varies tremendously, often depending upon the processing capability of the information handling system that supports the games. Many simple games are played on portable information handling systems, such as “apps” played on tablets and smartphones. Generally, these more simple games have graphical images that use less-intensive processing so that less-powerful portable information handling systems are capable of presenting the graphical images. Other more complex games are played on information handling systems built specifically for playing games, such as Xbox and Play Station. Specialized information handling systems often include graphics processing capabilities designed to support games with life-like graphical images presented on a high definition television or other type of display. Some of the most complex and life-like games are played on standardized information handling systems that include powerful processing components. For example, WINDOWS or LINUX operating systems run over an INTEL processor to execute a game and present images at one or more displays with the help of powerful graphics processing capabilities. Extreme gaming hardware, such as is available from ALIENWARE, presents life-like images with fluid and responsive movements based upon heavy processing performed by graphics processing units (GPUs).
Recently, a number of head-mounted display goggles have become available that support presentation of game images near the eyes of an end user. Typical head mounted displays fall into a virtual reality classification or an augmented reality classification. Virtual reality displays recreate a virtual world and block the ability to see elements in the real world. Virtual reality head-mounted displays mask the outside world with the goggles that cover the end user's eyes to provide a “separate” world, however, by masking the outside world such goggles prevent the end user from visually interacting with real world tools, such as keyboards, mice, stylus, LCDs, televisions, etc. . . . . For example, the Oculus Rift presents a game to an end user without letting the end user see outside of the goggles. Augmented reality displays typically maintain the real world context and provide bits of additional information to the user, however the information is typically presented in the periphery in a set location without knowledge of how to parse where the information is displayed. For example, the RECON JET provides a heads-up display for sports, such as cycling, triathlons and running.
Therefore a need has arisen for a system and method which augments visual images presented at a display with visual images presented by goggles.
In accordance with the present invention, a system and method are provided which substantially reduce the disadvantages and problems associated with previous methods and systems for presenting visual images. Real time compositing across multiple display elements augments a primary display of information as visual images with a secondary display of peripheral information as visual images relative to the primary display.
More specifically, an information handling system generates visual information that a graphics subsystem processes into pixel values to support presentation of visual images at a display and at goggles warn by an end user. The display and/or goggles include positional cues that cameras detect to determine the relative position of the goggles to the display. A compositing engine applies the relative position of the goggles to the display so that the display visual images and goggle visual images are presented in a desired relationship relative to each other. For example, the display visual images pass through the goggles at the position of the display relative to the goggles and the end user wearing the goggles so that the end user views the display visual images in a normal manner. Goggle visual information is generated only outside of the display visual information to create a composite display presentation in which the goggle visual information complements the display visual information. As the goggles move relative to the display, the composite visual presentation moves so that the display visual information shifts to adjust to changing alignments of the goggles and display.
The present invention provides a number of important technical advantages. One example of an important technical advantage is that goggles or other types of head gear allows an end user to have an enhanced visual experience by supplementing presentation of a primary display. A composited visual image is provided by allowing the end user to have direct viewing of an external display, such as a television, with peripheral images overlaid relative to the external display by the goggles while maintaining composition boundaries in real time during changing viewing angles, aspect ratios and distance. Compositing goggle images with peripheral display images is supported on a real time basis without end user inputs by automated use of reference points at the peripheral display that are detected by the goggles. In addition to providing see through portions of the goggles to support viewing of a peripheral display, reference points marked at other peripheral devices allow the end user to readily see and use physical devices not visible with virtual reality goggles that do not allow view outside of the goggle display.
The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the several figures designates a like or similar element.
Dynamically compositing visual images with goggles to supplement a primary display provides enhanced information handling system interactions. For purposes of this disclosure, an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, classify, process, transmit, receive, retrieve, originate, switch, store, display, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an information handling system may be a personal computer, a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, and a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.
Referring now to
Goggles 26 present visual information to an end user who wears the goggles proximate to his eyes. In the example embodiment, goggles 26 are managed by graphics subsystem 24 as a secondary display to external peripheral display 12, such as with display controls available from the WINDOWS operating system. Goggles 26 include a camera 38 that captures visual images along one or more predetermined axis, such as directly in front of goggles 26 or at the eyes of an end user wearing goggles 26 to monitor pupil movement. Goggles 26 include a microphone 40 that captures sounds made by an end user wearing goggles 26 and other nearby sounds, such as the output of speakers 46 interfaced with information handling system 10. Goggles 26 include an accelerometer 42 that detects accelerations of goggles 26 and a gyroscope that detects rotational movement of goggles 26, such as might be caused by an end user wearing goggles 26. The output of camera 38, microphone 40, accelerometer 42 and gyroscope 44 is provided to information handling system 10 through WPAN 30, and may also be used locally at goggles 26 to detect positional cues as set forth below. Display 12 includes an integrated camera 48 that captures images in front of display 12, such as images of an end user wearing goggles 26.
Information handling system 10 includes a compositing engine 50 that coordinates the presentation of visual images at display 12 and goggles 26 to provide an augmented virtual reality for an end user wearing goggles 26. Although compositing engine 50 is depicted as a firmware module executing in graphics subsystem 24, in alternative embodiments compositing engine 50 may execute as software on CPU 16, as firmware in goggles 26 or a distributed module having functional elements distributed between CPU 16, graphics subsystem 24 and goggles 26. Composting engine 50 determines the relative position of goggles 26 to display 12 and dynamically modifies visual images at goggles 26 to supplement the presentation of visual information at display 12. For example, goggles 26 frame the position of display 12 so that goggles 26 present information only outside of the frame. Alternatively, goggles 26 display one type of visual image outside the frame defined by the position of display 12 and another type of visual image within the frame of display 12, such as by blending images of goggles 26 and display 12 within the frame of display 12 and presenting a primary image with goggles 26 outside of the frame. In one alternative embodiment, a darkening surface within goggles 26 forms a frame around display 12 so that images from display 12 pass through goggles 26 but external light outside of the frame of display 12 does not pass through goggles 26.
Compositing engine 50 determines the relative position of goggles 26 to display 12 with a variety of positional cues detected by sensors at display 12 and goggles 26. One example of positional cues is physical position cue 52, which marks the physical housing boundary of display 12, such as with one or more infrared lights placed in the bezel of display 12 proximate but external to an LCD panel that presents images at display 12. Camera 38 in goggles 26 capture an image with physical positional cues 52 so that the viewing angle of goggles 26 relative to display 12 is provided by the angular position of physical positioning cues 52 in the capture image, and the distance between display 12 and goggles 26 is provided by the angle between the positioning cues 52. Another example of positioning cues is a display image positional cue presented in display 12 and captured by camera 38 of goggles 26. Compositing engine 50 manages the type of displayed positional cue 54 to provide optimized position detection. In one example embodiment, display image positional cue 54 is presented with infrared light not visible to an end user wearing goggles 26, and includes motion cues that indicate predictive information about motion expected at goggles 26. For example, an end user playing a game who is about to get attacked from the left side will likely respond with a rapid head movement to the left; display image positional cue may include an arrow that indicates likely motion to goggles 26 so that a processor of goggles 26 that dynamically adjusts images on goggles 26 will prepare to adapt to the rapid movement. Another example of positioning cues are physical positional cues 56 located on the exterior of goggles 26 and detected by camera 48 to indicated viewing angle and distance information. In various embodiments, various combinations of positioning cues may be used to provide positional information between goggles 26 and display 12.
In addition to visual positioning cues, other types of sensors may provide positioning cues to aid in the alignment of visual images of display 12 and goggles 26 in a desired manner. A position predictor 74 applies information sensed by other types of sensors to predict motion of goggles 26 relative to display 12. Sounds detected by microphone 40 may indicate an upcoming position change, such as by increased motion during times of excitement indicated by shouting, or stereoscopic indications that draw an end user's attention in a direction and cause the end user to look in that direction. Visual images of camera 38 taken of an end user's eyes may indicate an upcoming position change based upon dilation of pupils that indicate excitement of changes of eye direction that prelude head movements. In both examples, indications of increased excitement that are often preludes to head movement may be used to pre-allocate processing resources to more rapidly detect and react to movements when the movements occur. Accelerometer 42 and gyroscope 44 detect motion rapidly to provide predictive responses to movement before images from visual positional cues are analyzed and applied. Other positioning cues may be provided by an application as part of visual and audio presentation, such as visual images and sounds not detectable by an end user but detected by microphone 40 and camera 38 during presentation of the visual and audio information. Additional positioning cues may be provided by radio signal strength of goggles 26 for WPAN 30, which indicates a distance to display 12.
Referring now to
During operation, an end user puts goggles 26 on over his eyes and looks at display 12 through goggles 26. Compositing engine 50 generates visual images at display 12, such as images associated with a computer game. Compositing engine 50 determines the position of goggles 26 relative to display 12 based upon detection of positional cues by camera sensors, and allows visual images generated at display 12 to pass through external display zone 62, which corresponds to the position of display 12 relative to an end user wearing goggles 26. In one embodiment, goggles 26 do not present visual images in external display zone 62; in an alternative embodiment, goggles 26 present visual images in external display zone 62 that supplements visual images presented by display 12. Compositing engine 50 generates goggle visual information for presentation in goggle display zone 64. External display zone 62 changes position as an end user moves relative to display 12 so that external display 12 visual information passes through goggles 26 in different positions and goggle 26 visual information is presented in different areas. Movement of external display zone 62 is managed by compositing engine 50 in response to positional cues sensed by camera 38 and/or 48. Compositing engine 50 predicts the position of external display zone 62 between positional cue sensing operations by applying accelerometer and gyroscope sensed values, such as by a prediction of goggle 26 position changes based on detected accelerations and axis changes. In one embodiment, if an end user looks completely away from display 12, compositing engine 50 presents the information of display 12 as goggle visual information centered in goggles 26.
Referring now to
Although the present invention has been described in detail, it should be understood that various changes, substitutions and alterations can be made hereto without departing from the spirit and scope of the invention as defined by the appended claims.
