Patent Application
20140375682
The present invention relates generally to the field of virtual reality software and more particularly to content display controls.
Augmented reality (AR) provides a live, direct or indirect, view of a physical, real-world environment coupled, or augmented, with computer-generated content. The computer-generated content includes, but is not limited to: (i) sound; (ii) video; (iii) graphics; or (iv) other data intermixed while viewing the real world. The augmentation is overlaid in and around the real-time environment and may include semantic context. For instance, stock information might be displayed while viewing a corporation's signage, or player statistics might be displayed for a tennis player while watching a tennis match. As a result, the technology functions by enhancing one's current perception of reality.
Augmented reality rendering devices include, but are not limited to: (i) optical projection systems; (ii) monitors; (iii) handheld devices; (iv) and display systems worn on one's person, usually surrounding the head or in the field-of-view of one's eyes.
A head-mounted display (HMD) is one device that is worn as a helmet. HMDs place images of both the physical world and virtual content over the user's field-of-view. HMDs may employ sensors that allow the rendering system to align projected virtual content to the projected physical world.
Augmented reality content can be rendered on devices which in appearance are similar to conventional eyeglasses. These AR eyeglasses may employ cameras to intercept the real world view and re-display it coupled with augmented content, which is viewed through the eyepieces.
Embodiments of the present invention disclose a method, computer program product, and system for performing a computer action to manage a visual display on an augmented reality computing device. The method comprises receiving parameters representing a user command entered on at least one tactile sensor of an augmented reality computing device. One or more processors determine a computer action represented by the user command. The method further comprises modifying the display of content at a specific location on the augmented reality computing device, in response to determining the computer action.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code/instructions embodied thereon.
Any combination of computer-readable media may be utilized. Computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of a computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The present invention will now be described in detail with reference to the Figures. The following Figures provide an illustration of one embodiment. The embodiment, taken in part or in whole, does not imply any limitations with regard to the environments in which different embodiments may be implemented.
Server computer 140 may be a management server, a web server, or any other electronic device or computing system capable of receiving and sending data, and capable of communicating with devices, such as augmented reality computing device 130, via network 120. In other embodiments, server computer 140 may represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment.
In one embodiment, server computer 140 contains preference repository 160. Preference repository 160 holds user preferences that each represent a mapping between user commands and actions taken by augmented reality computing device 130. The actions taken by augmented reality computing device 130 include actions to modify the display of content at a specific location on augmented reality computing device 130. Examples of actions to modify the display of content at a specific location on augmented reality computing device 130 are shown in
Additionally, server computer 140 may contain other software, not shown in
Augmented reality computing device 130 is a computing system capable of displaying content to a user. For example, augmented reality computing device 130 may be eyeglasses with a right lens and a left lens, having the capability of projecting content in the field-of-view of a user. In general, augmented reality computing device 130 may be any device such as an optical projection system, monitor, handheld device, or display system worn on a user's person, or physically proximate to a user's person, capable of projecting content to the user. Augmented reality computing device 130 contains two software programs, UI control program 150 and sensor application programming interface (API) 170.
In one embodiment of the present invention, augmented reality computing device 130 contains a plurality of tactile sensors. The tactile sensors may be integrated: (i) within the assembly of augmented reality computing device 130; (ii) around the edge of augmented reality computing device 130; or (iii) in any location that would allow the user to access the sensor or interact with the sensor. A tactile sensor is a device that may be sensitive to touch, force, or pressure, light, or heat, for example. Tactile sensors may include piezoresistive, piezoelectric, capacitive and elastoresistive sensors, for example. A tactile sensor may receive and respond to a stimulus from a user's touch. A sensitivity of a sensor indicates how much the output of the sensor changes when the measured quantity changes. Tactile sensors can be deployed wherever interactions between a contact surface and a user are to be measured.
In one embodiment, the plurality of tactile sensors are configured to detect if a user is touching a respective sensor. Generally, data from the plurality of tactile sensors is accessed by a program, such as UI control program 150, by calling an Application Programming Interface (API), sensors API 170, provided with augmented reality computing device 130.
Sensors API 170 contains the instructions to interface with the plurality of tactile sensors described above. In one embodiment, the data from the plurality of tactile sensors is in a raw form. Sensors API 170 contains the instructions to transform the raw data into parameters representing a user command. Sensors API 170 provides UI control program 150 with the parameters representing the user command. In another embodiment, sensors API 170 minimally processes the raw data from the one or more tactile sensors. The computer instructions to translate the tactile sensor raw data into parameters representing a user command usable by UI control program 150 can be encapsulated within UI control program 150.
The parameters representing the user command transferred from sensors API 170 are interpreted by UI control program 150 using the mappings from preference repository 160 to determine a new action for augmented reality computing device 130; the new action is sent by UI control program 150 to augmented reality computing device 130. The actions taken by augmented reality computing device 130 include actions to modify the display of content at a specific location on augmented reality computing device 130. UI control program 150 is described in further detail in reference to
Embodiments of the present invention recognize that previous solutions do not supply sufficient control of the user experience, both from the perspective of user preference and user need. For example, a user might have diminished vision in one eye. Consequently, the user has a genuine need for the rendering of AR content on one frame, or section of the device, over another. Additionally, embodiments of the present invention recognize the lack of user feedback in the adaptation of learned preferences.
In step 210, UI control program 150 initializes user preferences from preference repository 160. In one embodiment, UI control program 150 communicates with server computer 140 to request specific information on user preferences from preference repository 160. A user of augmented reality computing device 130 will enter sign-in credentials, such as username, and in other embodiments, a password, when prompted by UI control program 150. This allows UI control program 150 to retrieve the corresponding user preferences. The sign-in methods include, but are not limited to: (i) default users; (ii) guests; and (iii) allowing device use without a sign-in procedure.
The method to communicate over a network, such as network 120, see
In step 220, UI control program 150 receives parameters representing a user command. In one embodiment, sensors API 170 processes raw tactile sensor data into parameters representing the user command and passes the parameters to UI control program 150. Examples of user commands represented by parameters are shown in
An alternative embodiment of the present invention involves sensors API 170 minimally processing the raw data from the one or more tactile sensors. The computer instructions to translate the tactile sensor raw data into parameters representing a user command usable by UI control program 150 can be encapsulated within UI control program 150. In this embodiment, sensors API 170 is essentially a conduit for raw data from the tactile sensors to the UI control program 150.
An alternative embodiment of the present invention combines sensors API 170 and UI control program 150 in such a manner as to eliminate the need of communications between the programs.
An alternative embodiment of the present invention, not shown in Figures, involves having sensors API 170 encapsulated outside augmented reality computer device 130. For instance, sensors API 170 can exists on server computer 140, or a computer similar to server computer 140, which is accessible via network 120.
An alternative embodiment of the present invention, not shown in Figures, involves having tactile sensor(s) encapsulated outside the assembly of augmented reality computing device 130. Tactile sensors can exist on computing devices that simulate tactile sensors, or tactile sensors can be located on mock eyewear. For instance, simulating tactile sensors is beneficial in a development environment or in a testing environment. For instance, simulated tactile sensors can exist on server computer 140, or a computer similar to server computer 140, which is accessible via network 120.
In step 230, UI control program 150 determines a computer action based on the parameters representing the user command received in step 220. In one embodiment, UI control program 150 queries preference repository 160 in order to determine a computer action to be taken by augmented reality computing device 130. Preference repository 160 returns the computer action back to UI control program 150 via network 120. For instance, a user taps on the right lens of augmented reality computing device 130 once. When the tap occurs UI control program 150 determines, by querying preference repository 160, that the computer action is to move the content displayed at specific locations in both lenses of augmented reality computing device 130 to new specific locations slightly to the left.
In step 240, UI control program 150 provides the computer action to augmented reality computing device 130. In one embodiment, UI control program 150 issues commands to augmented reality computing device 130. The commands issued by UI control program 150 can take a form that includes, but is not limited to: (i) extensible markup language (XML); (ii) variations of such, one being “beep” (Blocks Extensible Exchange Protocol); (iii) transmission control protocol/internet protocol (TCP/IP) or its derivatives; (iv) process communication, such as messaging; and (v) any communication commands that are to be developed for data handshaking.
In step 250, UI control program 150 updates preference repository 160. The user of the augmented reality computing device 130 has control of his or her user preferences. A user can build a new preference, update existing preferences, or delete preferences from preference repository 160. Updating preference repository 160 involves data handshaking with preference repository 160 in a similar fashion, as formerly described, in step 210. In one embodiment, augmented reality computing device 130 has a standard default sensor that indicates to UI control program 150 to enter a mode where, after one or more user interactions, UI control program 150 stores the user's preference. There are many derivative embodiments that would control user interactions; however, the purpose would be similar—stores the user's preference in the repository.
In another embodiment, in step 250, UI control program 150 may enter a learning mode. Once a computer action is executed, via the previous step 240, the computer action is logged in order determine possible patterns based on how many times, and in what context, the user wanted a specific computer action. UI control program 150 learns from repeated user commands for a specific computer action, heuristically analyzing the user commands, and noting any correlation between the type of computer action and in what context the computer action is executed. Context may include the type of content displayed on augmented reality computing device 130 during the computer action or other variables existing during the computer action.
Other variables may include any information that may be determined by the tactile sensors on augmented reality computing device 130 or any information that may be determined by other sensors (not shown). For example, augmented reality computing device may contain an accelerometer that can sense movement. In addition to sensed information, context variables may include the collection of location, ambient, or environmental data around the augmented reality computing device when a specific computer action is logged such as sound, perspective, light, darkness, focus, temperature, time of day, location of the user, or objects within the view of the lens view of the augmented reality device. For example, a specific computer action may be performed on the augmented reality device when a user is at work on a weekday that may be differentiated from when the user is in a home environment. The collection of location, ambient, or environmental data can optionally be provided by a user or a service on behalf of a user. These context variables are meant to aid in the intelligence and training of the user intent for invoking a specific computer action to modify the display of content at a specific location on an augmented reality computing device.
A threshold for the number of times a user command is repeated in order to invoke UI control program 150 learning can be established. For example, the threshold can be, but is not limited to: (i) more than once; (ii) more than once with one or more commands intermingled; (iii) once, if always executed in similar context, such as, the very first command when UI control program 150 is powered on; and (vi) when the user starts walking. Examples of such heuristically learned correlations include, but are not limited to: (i) user always moves text to one eye—system learns to display text on that eye; (ii) user moves content out of the way when walking—system always moves content out of the way, in one or both lens, when the user begins a walking movement; and (iii) user always enlarges small or detailed content in one lens, this might indicate that one eye is worse, thus system would either regularly: (a) enlarge detailed content in that lens, (b) always display it in the other lens, or (c) take similar resulting action.
The logging method (in step 250) to manage the logging data can take any form that would facilitate storage and retrieval of information. Examples of the logging method would include, but are not limited to: (i) utilizing AR environmental variables; (ii) utilizing a database management system (DBMS) such as: (a) a relational database, (b) hierarchical database, (c) object-oriented database, (d) XML (Extendable Markup Language) database, etc.; (iii) utilizing a flat file; (iv) utilizing a table lookup scheme, such as a hash table software; or (v) utilizing any custom or off-the-shelf software that would manage the logging data. Additionally, the learned activity is stored in another repository similar to preference repository 160. Updating the learned activity involves communicating with the repository in a similar fashion, as formerly described, in step 210.
In another embodiment, in step 250 of UI control program 150, where UI control program 150 enters a heuristic learning mode, step 250 is implemented as a self-contained program. This embodiment of the present invention would perform all the functions described formerly in step 250. The embodiment would include a method of data handshaking of data between augmented reality computing device 130 and this alternative embodiment.
In decision 260, UI control program 150 determines if any further processing is necessary and if not UI control program 150 terminates. Step 260's implementation depends upon program implementation of augmented reality computing device 130, as someone skilled in the art would recognize. Examples of 260's implementation would include, but are not limited to: (i) simple termination; (ii) sleeping until interrupted; (iii) a signal shutdown; (iv) wait until a time event; (v) wait until a user event; (vi) loop forever; (vii) countdown loop; or (viii) any combination of such.
In an alternative embodiment of the present invention the tactile sensor is in the assembly of augmented reality computing device 130. More specifically, the tactile senor is in one or both of the extending side-arms of augmented reality computing device 130 (not labeled, but shown in
In an alternative embodiment of the present invention the lens-of-focus is highlighted as an indication to the user of the context in which he is manipulating the AR content. The lens-of-focus is the lens in which the user's intended computer action will be executed. For instance, in
In
Preference repository 160 is an information store of user's preferences for augmented reality computing device 130. The information store shown in
Example mappings from user-commands 420 to actions 430 are shown in
Additional examples of mappings according to embodiments of the present invention, from user-commands 420 to actions 430, which are not shown in
Preferences repository 160 can be loaded, fully or partially, (i) into memory accessible by the UI control program 150 or (ii) continuously queried by UI control program 150. Such implementation differences would affect the data handshaking method described formerly in step 210.
Augmented reality computing device 130 and server computer 140 each include communications fabric 502, which provides communications between computer processor(s) 504, memory 506, persistent storage 508, communications unit 510, and input/output (I/O) interface(s) 512. Communications fabric 502 can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric 502 can be implemented with one or more buses.
Memory 506 and persistent storage 508 are computer-readable storage media. In this embodiment, memory 506 includes random access memory (RAM) 514 and cache memory 516. In general, memory 506 can include any suitable volatile or non-volatile computer-readable storage media.
Sensors API 170 and UI control program 150 are stored in persistent storage 508 of augmented reality computing device 130 for execution and/or access by one or more of the respective computer processors 504 of augmented reality computing device 130 via one or more memories of memory 506 of augmented reality computing device 130. Preference repository 160 is stored in persistent storage 508 of server computer 140 for execution and/or access by one or more of the respective computer processors 504 of server computer 140 via one or more memories of memory 506 of server computer 140. In this embodiment, persistent storage 508 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 508 can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage media that is capable of storing program instructions or digital information.
The media used by persistent storage 508 may also be removable. For example, a removable hard drive may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 508.
Communications unit 510, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit 510 includes one or more network interface cards. Communications unit 510 may provide communications through the use of either or both physical and wireless communications links. Sensors API 170 and UI control program 150 may be downloaded to persistent storage 508 of augmented reality computing device 130 through communications unit 510 of augmented reality computing device 130. Preference repository 160 may be downloaded to persistent storage 508 of server computer 140 through communications unit 510 of server computer 140.
I/O interface(s) 512 allows for input and output of data with other devices that may be connected to augmented reality computing device 130 or server computer 140. For example, I/O interface 512 may provide a connection to external devices 518 such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices 518 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, e.g., sensors API 170 and UI control program 150, can be stored on such portable computer-readable storage media and can be loaded onto persistent storage 508 of augmented reality computing device 130 via I/O interface(s) 512 of augmented reality computing device 130. Software and data used to practice embodiments of the present invention, e.g., preference repository 160, can be stored on such portable computer-readable storage media and can be loaded onto persistent storage 508 of server computer 140 via I/O interface(s) 512 of server computer 140. I/O interface(s) 512 also connect to a display 520.
Display 520 provides a mechanism to display data to a user and may be, for example, a computer monitor.
The programs described herein are identified based upon the application for which they are implemented in a specific embodiment of the invention. However, it should be appreciated that any particular program nomenclature herein is used merely for convenience, and thus the invention should not be limited to use solely in any specific application identified and/or implied by such nomenclature.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
