Many computing applications such as computer games, multimedia applications, or the like use controls to allow users to manipulate cursors, game characters, or other aspects of an application. Today, designers and engineers in the area of consumer devices, such as computers, televisions, DVRs, game consoles, and appliances, have many options for user-device interaction with a cursor. Input techniques may leverage a remote control, keyboard, mouse, stylus, game controller, touch, voice, gesture, and the like. For example, an image capture device can detect user gestures for controlling a cursor. For any given technique, the design of user interface feedback is critical to help users interact more effectively and efficiently with the device.
One of the most well-known input mechanisms and interaction feedback designs is the mouse and on-screen cursor. The design of each has evolved and been refined over many years. In addition, on-screen cursor feedback has even been decoupled from the mouse and applied to other forms of user input where targeting on-screen objects, such as buttons, or other elements is essential to avoid user frustration.
Effective targeting and other gestural interactions using a cursor require real-time user interface feedback indicating the cursor's position to the user. However, displaying a traditional cursor graphic, such as an arrow, at the exact position of the cursor suffers from a variety of disadvantages. In a real-world gestural system, where lag and jitter are difficult to avoid and reliable cursor control requires use of more sophisticated targeting assistance techniques, the disadvantages of displaying a graphic at the precise position of the cursor are magnified. The cursor precision suggested by such a graphic and consequently expected by the user is poorly matched with the realities of the system.
Accordingly, it is desirable to provide systems and methods for improving user interface feedback regarding cursor position on a display screen of an audiovisual device.
Disclosed herein are systems and methods for providing user interface feedback regarding a cursor position on a display screen of an audiovisual device. According to one embodiment, a user may use a suitable input device for controlling a cursor in a computing environment. The actual position of the cursor may not be displayed on a display screen in the computing environment. In other words, the cursor's actual position may be hidden from the user's view. However, in accordance with the presently disclosed subject matter, the displayed objects may provide feedback regarding the cursor's position. Particularly, a position of the cursor may be compared to an object's position for determining whether the cursor is positioned on the display screen at the same position as a portion of the object or within a predetermined distance of the portion of the object. In response to determining the cursor is positioned on the display screen at the same position as the portion of the object or within the predetermined distance of the portion of the object, an appearance of the portion of the object may be altered. For example, brightness, color, or other appearance of the portion of the object may be altered for indicating to the user that the cursor's position is near the portion of the object. These techniques and other disclosed herein can be advantageous in gestural systems, for example, or other systems for overcoming difficulties of lag and jitter and unreliable cursor control.
In another embodiment of the subject matter disclosed herein, a plurality of objects displayed on a display screen may be utilized for providing user feedback regarding a cursor's position. The cursor's position with respect to the objects' positions may be determined. Particularly, it is determined whether the cursor is positioned on the display screen at the same position as one or more of the objects or within a predetermined distance of one or more of the objects. Input from the user for controlling movement of the cursor is received. In response to the user control of the cursor, an appearance of one or more of the objects is altered if the cursor is positioned on the display screen at the same position as the object(s) or within a predetermined distance of the object(s). Further, in response to the user control of the cursor, one or more of the objects can move if the cursor is positioned on the display screen at the same position as the object(s) or within a predetermined distance of the object(s). Accordingly, one or more objects may move or the objects' appearance may change in response to user control of the cursor based on the cursor's proximity to the object(s).
In yet another embodiment of the subject matter disclosed herein, user input is received in a computing environment based on cursor position. Particularly, a cursor's position with respect to a display screen is determined when the cursor is positioned off of the display screen. For example, as opposed to the cursor being positioned within a display screen, a computing environment may track a cursor's position when the cursor has moved outside of the bounds of the display screen. A user may move the cursor off of the display screen and continue to move the cursor outside the bounds of the display screen. This movement may be tracked by the computing environment, and the cursor's position stored in memory. While the cursor's position is outside the bounds of the display screen, a direction of the cursor's position with respect to the display screen may be indicated such as, for example, by a displayed object. The positioning of the displayed object may be adjacent or otherwise near a side of the display screen that is closest to the cursor's position for indicating that the cursor's position is in that direction with respect to the display screen. In response to the user's control of the cursor when the cursor is positioned off of the display screen, an element, or another object on the display screen, may be controlled based on the cursor's position. For example, one or more objects on the display screen may be manipulated (e.g., rotated or otherwise moved) based on movement of the cursor off of the display screen. In this way, even though the cursor's position is not on the display screen, movement of the cursor may control displayed elements.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
The systems, methods, and computer readable media for altering a view perspective within a virtual environment in accordance with this specification are further described with reference to the accompanying drawings in which:
As will be described herein, user interface feedback may be provided regarding a cursor position on a display screen of an audiovisual device. According to one embodiment, a user may use gestures, a mouse, a keyboard, or the like to control a cursor in a computing environment. The actual position of the cursor may not be displayed on a display screen in the computing environment, such as by use of an arrow-shaped object to show the cursor's exact position; however, in accordance with the presently disclosed subject matter, the one or more displayed objects may provide feedback regarding the cursor's position. Particularly, a position of the cursor may be compared to an object's position for determining whether the cursor is positioned on the display screen at the same position as a portion of the object or within a predetermined distance of the portion of the object. In response to determining the cursor is positioned on the display screen at the same position as the portion of the object or within the predetermined distance of the portion of the object, an appearance of the portion of the object may be altered. For example, brightness, color, or other appearance of the portion of the object may be altered for indicating to the user that the cursor's position is near the portion of the object.
In another embodiment of the subject matter disclosed herein, a plurality of objects displayed on a display screen may be utilized for providing user feedback regarding a cursor's position. The cursor's position with respect to the objects' positions may be determined. Particularly, it is determined whether the cursor is positioned on the display screen at the same position as one or more of the objects or within a predetermined distance of one or more of the objects. Input from the user for controlling movement of the cursor is received. In response to the user control of the cursor, an appearance of one or more of the objects is altered if the cursor is positioned on the display screen at the same position as the object(s) or within a predetermined distance of the object(s). Further, in response to the user control of the cursor, one or more of the objects can move if the cursor is positioned on the display screen at the same position as the object(s) or within a predetermined distance of the object(s). Accordingly, one or more objects may move or the objects' appearance may change in response to user control of the cursor based on the cursor's proximity to the object(s).
In yet another embodiment of the subject matter disclosed herein, user input is received in a computing environment based on cursor position. Particularly, a cursor's position with respect to a display screen is determined when the cursor is positioned off of the display screen. For example, as opposed to the cursor being positioned within a display screen, a computing environment may track a cursor's position when the cursor has moved outside of the bounds of the display screen. A user may move the cursor off of the display screen and continue to move the cursor outside the bounds of the display screen. This movement may be tracked by the computing environment, and the cursor's position stored in memory. While the cursor's position is outside the bounds of the display screen, a direction of the cursor's position with respect to the display screen may be indicated such as, for example, by a displayed object. The positioning of the displayed object may be adjacent or otherwise near a side of the display screen that is closest to the cursor's position for indicating that the cursor's position is in that direction with respect to the display screen. In response to the user's control of the cursor when the cursor is positioned off of the display screen, an element, or another object on the display screen, may be controlled based on the cursor's position. For example, one or more objects on the display screen may be manipulated (e.g., rotated or otherwise moved) based on movement of the cursor off of the display screen. In this way, even though the cursor's position is not on the display screen, movement of the cursor may still control displayed elements.
A user may control a cursor's position by using any number of suitable user input devices such as, for example, a mouse, a trackball, a keyboard, an image capture device, or the like. A user may control a cursor displayed in a computing environment such as a game console, a computer, or the like. In an example of controlling a cursor's position, a mouse may be moved over a surface for controlling a direction of movement, speed of movement, positioning of a cursor on and off of a display screen, and the like. In yet another example, the keys of a keyboard (e.g., the direction arrow keys) may be configured for controlling the cursor.
In an exemplary embodiment, user gestures may be detected by, for example, an image capture device. For example, the capture device may capture a depth image of a scene including a user. In one embodiment, the capture device may determine whether one or more targets or objects in the scene correspond to a human target such as the user. If the capture device determines that one or more objects in the scene is a human, it may determine the depth to the human as well as the size of the human. The device may then center a virtual screen around each human target based on stored information, such as, for example a look up table that matches size of the person to wingspan and/or personal profile information. Each target or object that matches the human pattern may be scanned to generate a model such as a skeletal model, a mesh human model, or the like associated therewith. The model may then be provided to the computing environment such that the computing environment may track the model, determine which movements of the model are inputs for controlling an activity of a cursor, and render the cursor's activity based on the control inputs. Accordingly, the user's movements can be tracked by the capture device for controlling a direction of movement, speed of movement, positioning of a cursor on and off of a display screen, and the like.
An audiovisual device may be any type of display, such as a television, a monitor, a high-definition television (HDTV), or the like that may provide game or application visuals and/or audio to a user. For example, a computing environment may include a video adapter such as a graphics card and/or an audio adapter such as a sound card that may provide audiovisual signals associated with the game application, non-game application, or the like. The audiovisual device may receive the audiovisual signals from the computing environment and may then output the game or application visuals and/or audio associated with the audiovisual signals to the user. For example, a user may control a user input device for inputting control information for controlling or altering objects displayed on the display screen based on cursor positioning in accordance with the subject matter disclosed herein. According to one embodiment, the audiovisual device may be connected to the computing environment via, for example, an S-Video cable, a coaxial cable, an HDMI cable, a DVI cable, a VGA cable, or the like.
At 12, an object may be displayed on a display screen.
At 14 of
At 16 of
At 18 of
It should be noted that the appearances of a plurality of portions of the same object and/or portions of other objects can be simultaneously altered due to the cursor's position. In the particular example of
The appearance of an object or a portion of the object may be altered by changing its brightness, its color, or the like. Although in the example of
Objects 21-24 can be configured for selection for user input when the cursor is positioned on the display screen 20 at the same position as the object. When the cursor is at the same position as the object, the object can receive focus such that it can receive user input. An example is the case when a cursor is over an object, such as a button, that can be selected for input associated with the object when the cursor is on the object and one of the mouse buttons is clicked. In another example, the cursor's position can provide lighting and/or shadows on an avatar when in proximity to the avatar. In the depicted example, the object 21 has received focus, and this is indicated by a border 31 surrounding the object 21. The other objects 22-24 can also receive focus when the cursor's position is at the object.
At 34, a plurality of objects may be displayed on a display screen. For example,
At 36, it is determined whether the cursor is positioned on the display screen 20 at the same position as one or more of the objects 40 and 42, or within a predetermined distance of one or more of the objects 40 and 42. For example, in
At 38, responsive to user control of the cursor, an appearance of the objects is altered, or the objects are moved, if the cursor is positioned on the display screen at the same position as the objects or within a predetermined distance of the objects. For example, in
The objects can be distinguished based on their sizes, color, and/or the like for indicating the cursor's exact position. For example, referring to
At 52, a cursor's position with respect to a display screen may be determined when the cursor is positioned off of the display screen. For example, a computer may be configured to recognize when the cursor is positioned off of the display screen. In addition, the computer may track a distance, direction, and the like of the movement of the cursor while the cursor is positioned off of the display screen. For example, a mouse movement or gesture of a user's body while the cursor is off of the display screen may be tracked, and the cursor's position off of the display screen moved in accordance with the tracked movement.
At 54, a direction of the cursor's position with respect to the display screen is indicated. For example, one or more objects, such as the object 40 and 42 shown in
At 56, responsive to user control of the cursor when the cursor is positioned off of the display screen, one or more elements on the display screen may be controlled based on the cursor's position. For example, a distance and/or direction of movement of a cursor or a user's body part may be tracked when the cursor is off of the display screen, and a characteristic of an element may be altered based on the distance or direction of movement of the mouse or user's body part. In an example of altering a characteristic of an element on the display screen, the element may be an object that is rotated based on the cursor movement. In other examples, sound, other displayed features of objects, such as colors, brightness, orientation in space, and the like may be altered based on the cursor movement off of the display screen.
By varying the user interface feedback provided by the hidden or invisible cursor along multiple dimensions, the system can further engage the user and create a rich and playful experience for the user. For example, the intensity of the lighting when the cursor acts as a light source as described herein may be modified according to the intensity of the user's interaction, with faster gestures or mouse movements resulting in brighter or differently colored user interface feedback. Similarly, the cursor can interact with various user interface controls in different ways, suggesting materials with different physical properties. The behavior of the cursor can also be themed or personalized, so that one user's cursor interaction affecting a particular region of the display screen will see a different effect than another user's cursor interaction affecting the same region.
In a gesture-based system, the objects may provide additional feedback beyond cursor control. While passive during targeting gestures, the objects may react to symbolic or manipulative gestures, clarifying the mode of interaction and/or providing real-time feedback while the user is executing a gesture.
In another example, a cursor's position may cause alteration of the appearance of normally inactive objects or other features displayed, or hidden, on a display screen. If the cursor's position is at, or within a predetermined distance, of one or more of the inactive objects the appearance of the entire object, a portion of the object, and/or surrounding area, hidden or visible to a viewer, can be altered for indicating the proximity of the cursor's position. For example, a portion of a wallpaper or background image on a display screen may be altered based on the proximity of a cursor.
A graphics processing unit (GPU) 108 and a video encoder/video codec (coder/decoder) 114 form a video processing pipeline for high speed and high resolution graphics processing. Data is carried from the graphics processing unit 108 to the video encoder/video codec 114 via a bus. The video processing pipeline outputs data to an A/V (audio/video) port 140 for transmission to a television or other display. A memory controller 110 is connected to the GPU 108 to facilitate processor access to various types of memory 112, such as, but not limited to, a RAM (Random Access Memory).
The multimedia console 100 includes an I/O controller 120, a system management controller 122, an audio processing unit 123, a network interface controller 124, a first USB host controller 126, a second USB controller 128 and a front panel I/O subassembly 130 that are preferably implemented on a module 118. The USB controllers 126 and 128 serve as hosts for peripheral controllers 142(1)-142(2), a wireless adapter 148, and an external memory device 146 (e.g., flash memory, external CD/DVD ROM drive, removable media, etc.). The network interface 124 and/or wireless adapter 148 provide access to a network (e.g., the Internet, home network, etc.) and may be any of a wide variety of various wired or wireless adapter components including an Ethernet card, a modem, a Bluetooth module, a cable modem, and the like.
System memory 143 is provided to store application data that is loaded during the boot process. A media drive 144 is provided and may comprise a DVD/CD drive, hard drive, or other removable media drive, etc. The media drive 144 may be internal or external to the multimedia console 100. Application data may be accessed via the media drive 144 for execution, playback, etc. by the multimedia console 100. The media drive 144 is connected to the I/O controller 120 via a bus, such as a Serial ATA bus or other high speed connection (e.g., IEEE 1394).
The system management controller 122 provides a variety of service functions related to assuring availability of the multimedia console 100. The audio processing unit 123 and an audio codec 132 form a corresponding audio processing pipeline with high fidelity and stereo processing. Audio data is carried between the audio processing unit 123 and the audio codec 132 via a communication link. The audio processing pipeline outputs data to the A/V port 140 for reproduction by an external audio player or device having audio capabilities.
The front panel I/O subassembly 130 supports the functionality of the power button 150 and the eject button 152, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of the multimedia console 100. A system power supply module 136 provides power to the components of the multimedia console 100. A fan 138 cools the circuitry within the multimedia console 100.
The CPU 101, GPU 108, memory controller 110, and various other components within the multimedia console 100 are interconnected via one or more buses, including serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include a Peripheral Component Interconnects (PCI) bus, PCI-Express bus, etc.
When the multimedia console 100 is powered ON, application data may be loaded from the system memory 143 into memory 112 and/or caches 102, 104 and executed on the CPU 101. The application may present a graphical user interface that provides a consistent user experience when navigating to different media types available on the multimedia console 100. In operation, applications and/or other media contained within the media drive 144 may be launched or played from the media drive 144 to provide additional functionalities to the multimedia console 100.
The multimedia console 100 may be operated as a standalone system by simply connecting the system to a television or other display. In this standalone mode, the multimedia console 100 allows one or more users to interact with the system, watch movies, or listen to music. However, with the integration of broadband connectivity made available through the network interface 124 or the wireless adapter 148, the multimedia console 100 may further be operated as a participant in a larger network community.
When the multimedia console 100 is powered ON, a set amount of hardware resources are reserved for system use by the multimedia console operating system. These resources may include a reservation of memory (e.g., 16 MB), CPU and GPU cycles (e.g., 5%), networking bandwidth (e.g., 8 kbs), etc. Because these resources are reserved at system boot time, the reserved resources do not exist from the application's view.
In particular, the memory reservation preferably is large enough to contain the launch kernel, concurrent system applications and drivers. The CPU reservation is preferably constant such that if the reserved CPU usage is not used by the system applications, an idle thread will consume any unused cycles.
With regard to the GPU reservation, lightweight messages generated by the system applications (e.g., popups) are displayed by using a GPU interrupt to schedule code to render popup into an overlay. The amount of memory required for an overlay depends on the overlay area size and the overlay preferably scales with screen resolution. Where a full user interface is used by the concurrent system application, it is preferable to use a resolution independent of application resolution. A scaler may be used to set this resolution such that the need to change frequency and cause a TV resynch is eliminated.
After the multimedia console 100 boots and system resources are reserved, concurrent system applications execute to provide system functionalities. The system functionalities are encapsulated in a set of system applications that execute within the reserved system resources described above. The operating system kernel identifies threads that are system application threads versus gaming application threads. The system applications are preferably scheduled to run on the CPU 101 at predetermined times and intervals in order to provide a consistent system resource view to the application. The scheduling is to minimize cache disruption for the gaming application running on the console.
When a concurrent system application requires audio, audio processing is scheduled asynchronously to the gaming application due to time sensitivity. A multimedia console application manager (described below) controls the gaming application audio level (e.g., mute, attenuate) when system applications are active.
Input devices (e.g., controllers 142(1) and 142(2)) are shared by gaming applications and system applications. The input devices are not reserved resources, but are to be switched between system applications and the gaming application such that each will have a focus of the device. The application manager preferably controls the switching of input stream, without knowledge the gaming application's knowledge and a driver maintains state information regarding focus switches. The cameras 27, 28 and capture device 20 may define additional input devices for the console 100.
In
The computer 241 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
The computer 241 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 246. The remote computer 246 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 241, although only a memory storage device 247 has been illustrated in
When used in a LAN networking environment, the computer 241 is connected to the LAN 245 through a network interface or adapter 237. When used in a WAN networking environment, the computer 241 typically includes a modem 250 or other means for establishing communications over the WAN 249, such as the Internet. The modem 250, which may be internal or external, may be connected to the system bus 221 via the user input interface 236, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 241, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
It should be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered limiting. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or the like. Likewise, the order of the above-described processes may be changed.
Additionally, the subject matter of the present disclosure includes combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or processes disclosed herein, as well as equivalents thereof.