Patent Application
20230315247
The present disclosure relates generally to computer systems that are in communication with a display generation component and, optionally, one or more input devices that provide computer-generated experiences, including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via a display.
The development of computer systems for augmented reality has increased significantly in recent years. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices, such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch-screen displays for computer systems and other electronic computing devices are used to interact with virtual/augmented reality environments. Example virtual elements include virtual objects, such as digital images, video, text, icons, and control elements such as buttons and other graphics.
Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment, and systems in which manipulation of virtual objects are complex, tedious, and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment. In addition, these methods take longer than necessary, thereby wasting energy of the computer system. This latter consideration is particularly important in battery-operated devices.
Accordingly, there is a need for computer systems with improved methods and interfaces for providing computer-generated experiences to users that make interaction with the computer systems more efficient and intuitive for a user. Such methods and interfaces optionally complement or replace conventional methods for providing extended reality experiences to users. Such methods and interfaces reduce the number, extent, and/or nature of the inputs from a user by helping the user to understand the connection between provided inputs and device responses to the inputs, thereby creating a more efficient human-machine interface.
The above deficiencies and other problems associated with user interfaces for computer systems are reduced or eliminated by the disclosed systems. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is portable device (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch, or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a “touch screen” or “touch-screen display”). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generation component, the output devices including one or more tactile output generators and/or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory and one or more modules, programs or sets of instructions stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through a stylus and/or finger contacts and gestures on the touch-sensitive surface, movement of the user’s eyes and hand in space relative to the GUI (and/or computer system) or the user’s body as captured by cameras and other movement sensors, and/or voice inputs as captured by one or more audio input devices. In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, spreadsheet making, game playing, telephoning, video conferencing, e-mailing, instant messaging, workout support, digital photographing, digital videoing, web browsing, digital music playing, note taking, and/or digital video playing. Executable instructions for performing these functions are, optionally, included in a transitory and/or non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors.
There is a need for electronic devices with improved methods and interfaces for interacting with a computer system that is displaying one or more portions of a three-dimensional environment. Such methods and interfaces may complement or replace conventional methods for interacting with a computer system that is displaying one or more portions of a three-dimensional environment. Such methods and interfaces reduce the number, extent, and/or the nature of the inputs from a user and produce a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges.
In some embodiments, a computer system displays, via a first display generation component, a first user interface object while a first view of a three-dimensional environment is visible, wherein the first user interface object is displayed at a first position in the three-dimensional environment, and wherein the first position in the three-dimensional environment has a first spatial arrangement relative to a respective portion of a user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via one or more input devices, an input that corresponds to movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system maintains display of the first user interface object at a respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user while a second view of the three-dimensional environment is visible. While the second view of the three-dimensional environment is visible and while displaying the first user interface object in the second view of the three-dimensional environment, the computer system detects, via the one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the first gaze input directed to the first user interface object, in accordance with a determination that the first gaze input satisfies attention criteria with respect to the first user interface object, the computer system displays a plurality of affordances for accessing system functions of the first computer system.
In some embodiments, a computer system displays, via a first display generation component, a first user interface object while a first view of a three-dimensional environment is visible, wherein the first user interface object is displayed at a first position in the three-dimensional environment, and wherein the first position in the three-dimensional environment has a first spatial arrangement relative to a respective portion of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the first gaze input directed to the first user interface object: in accordance with a determination that a first event for a first notification satisfies timing criteria, the computer system displays content associated with the first notification; and in accordance with a determination that the first event for the first notification does not satisfy the timing criteria, the computer system displays a plurality of affordances for performing system operations associated with the first computer system without displaying the content associated with the first notification.
In some embodiments, a computer system displays, via a first display generation component, a first user interface object while a first view of a three-dimensional environment is visible, wherein the first user interface object is displayed at a first position in the three-dimensional environment, and wherein the first position in the three-dimensional environment has a first spatial arrangement relative to a respective portion of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the first gaze input directed to the first user interface object: in accordance with a determination that there is a request for the first computer system to join a first communication session that satisfies timing criteria, the computer system displays a first user interface that includes an affordance for joining the first communication session; and in accordance with a determination that there is not a request to join a communication session that satisfies the timing criteria, the computer system displays a plurality of affordances for performing system operations associated with the first computer system without displaying the affordance for joining a communication session.
In some embodiments, a computer system displays, via a first display generation component, a first view of a three-dimensional environment that corresponds to a first viewpoint of a user. The first view of the three-dimensional environment includes a first user interface element that is displayed at a first position in the three-dimensional environment, the first user interface element has a first spatial relationship with the first viewpoint of the user while the first user interface element is displayed at the first position in the three-dimensional environment, and the first user interface element includes at least a first affordance. While displaying the first view of the three-dimensional environment, including displaying the first user interface element at the first position in the three-dimensional environment, the computer system detects first movement of a viewpoint of the user from the first viewpoint to a second viewpoint. In response to detecting the first movement of the viewpoint of the user from the first viewpoint to the second viewpoint, while a second view of the three-dimensional environment that corresponds to the second viewpoint of the user is visible via the one or more display generation components, the computer system displays the first user interface element at a second position in the three-dimensional environment, wherein the first user interface element has the first spatial relationship with the second viewpoint of the user while the first user interface element is displayed at the second position in the three-dimensional environment. While displaying the first user interface element in the second view of the three-dimensional environment, the computer system detects, via one or more input devices, a first input that corresponds to activation of the first affordance. In response to detecting the first input that corresponds to activation of the first affordance, the computer system displays a second user interface element in the second view of the three-dimensional environment, wherein the second user interface element is displayed at a third position in the three-dimensional environment. While displaying the second view of the three-dimensional environment, including displaying the second user interface element at the third position in the three-dimensional environment, the computer system detects second movement of the viewpoint of the user from the second viewpoint to a third viewpoint. In response to detecting the second movement of the viewpoint of the user from the second viewpoint to the third viewpoint, while a third view of the three-dimensional environment that corresponds to the third viewpoint of the user is visible via the one or more display generation components, the computer system displays the second user interface element at a location in the third view of the three-dimensional environment that corresponds to the third position in the three-dimensional environment.
In some embodiments, while a first view of a three-dimensional environment is visible via the first display generation component, a computer system detects, via one or more input devices, a first user input, including detecting a first gaze input that is directed to a first position in the three-dimensional environment. In response to detecting the first user input including detecting the first gaze input: in accordance with a determination that the first position in the three-dimensional environment has a first spatial relationship to a viewport through which the three-dimensional environment is visible, the computer system displays a first user interface object in the first view of the three-dimensional environment, wherein the first user interface object includes one or more affordances for accessing a first set of functions of the first computer system, wherein the first user interface object is displayed at a second position in the three-dimensional environment that has a second spatial relationship, different from the first spatial relationship, to the viewport through which the three-dimensional environment is visible; and in accordance with a determination that the first position in the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment is visible, the computer system forgoes displaying the first user interface object in the first view of the three-dimensional environment.
In some embodiments, while a first view of a three-dimensional environment is visible via the first display generation component, a computer system detects, via one or more input devices, that attention of a user of the computer system is directed to a first portion of the first view of the three-dimensional environment. In response to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment: in accordance with a determination that the first portion of the first view of the three-dimensional environment has a first spatial relationship to a viewport through which the three-dimensional environment is visible while a first set of one or more contextual conditions are met at a time that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, the computer system displays a first set of one or more user interface objects that correspond to the first set of one or more contextual conditions; and in accordance with a determination that the first portion of the first view of the three-dimensional environment has the first spatial relationship to the viewport through which the three-dimensional environment is visible while a second set of one or more contextual conditions, different from the first set of one or more contextual conditions, are met at the time that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, the computer system displays a second set of one or more user interface objects that correspond to the second set of one or more contextual conditions, the second set of one or more user interface objects being different from the first set of one or more user interface objects.
In some embodiments, while a first view of an environment is visible via one or more display generation components, a computer system detects occurrence of a first event. In response to detecting the occurrence of the first event, the computer system displays, via a first display generation component, a first indication of a first notification corresponding to the first event. After displaying, via the first display generation component, the first indication of the first notification corresponding to the first event: in accordance with a determination that attention of a user of the computer system was directed to the first indication within a first threshold amount of time and then moved away from the first indication before first criteria are met, the computer system maintains display of the first indication for a first duration of time and ceases display of the first indication after expiration of the first duration of time; and in accordance with a determination that the attention of the user of the computer system was not directed to the first indication within the first threshold amount of time, the computer system maintains display of the first indication for a second duration of time and ceases display of the first indication after expiration of the second duration of time, wherein the second duration of time is different from the first duration of time.
In some embodiments, while a respective view of a three-dimensional environment that includes one or more physical objects is visible via a first display generation component, a computer system detects a first input that corresponds to a request to display one or more notifications in the respective view of the three-dimensional environment. In response to detecting the first input, the computer system displays a respective notification that corresponds to a previously detected event at a respective location in the three-dimensional environment. In accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a first set of values for a first visual characteristic, the computer system displays the respective notification with a first visual appearance that is based at least in part on the first set of values for the first visual characteristic of the representation of the physical environment at the respective location; and in accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a second set of values, different from the first set of values, for the first visual characteristic, the computer system displays the respective notification with a second visual appearance, different from the first visual appearance, that is based at least in part on the second set of values for the first visual characteristic of the representation of the physical environment at the respective location.
In some embodiments, a computer system displays, via the first display generation component, a first user interface object in a first view of a three-dimensional environment that corresponds to a first viewpoint of a user, wherein the first user interface object is displayed at a first position in the three-dimensional environment and has a first spatial relationship with the first viewpoint of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system, via the one or more input devices, first movement of a current viewpoint of the user from the first viewpoint to a second viewpoint of the user. In response to detecting the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint, the computer system: replaces display of the first view of the three-dimensional environment with display of a second view of the three-dimensional environment that corresponds to the second viewpoint of the user; in accordance with a determination that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet first criteria and that the first user interface object is of a first type, displays the first user interface object in the second view of the three-dimensional environment, wherein the first user interface object is displayed at a second position, different from the first position, in the three-dimensional environment and has the first spatial relationship with the second viewpoint of the user; and in accordance with a determination that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet the first criteria and that the first user interface object is of a second type different from the first type, forgoes displaying the first user interface object at the second position in the three-dimensional environment in the second view of the three-dimensional environment. The computer system detects via the one or more input devices, second movement of the current viewpoint of the user from the second viewpoint to a third viewpoint of the user. In response to detecting the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint, the computer system: replaces display of the second view of the three-dimensional environment with display of a third view of the three-dimensional environment that corresponds to the third viewpoint of the user and in accordance with a determination that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria, displays the first user interface object in the third view of the three-dimensional environment, wherein the first user interface object is displayed at a third position, different from the first position and the second position, in the three-dimensional environment and has the first spatial relationship with the third viewpoint of the user, irrespective of whether the first user interface object is of the first type or the second type.
Note that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.
For a better understanding of the various described embodiments, reference should be made to the Description of Embodiments below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.
The present disclosure relates to user interfaces for providing an extended reality (XR) experience to a user, in accordance with some embodiments.
The systems, methods, and GUIs described herein improve user interface interactions with virtual/augmented reality environments in multiple ways.
In some embodiments, a computer system displays a plurality of affordances for performing system functions (e.g., system settings such as audio volume, a centralized notifications user interface, a search function, a virtual assistant) of the computer system in response to a user directing a gaze input to a first user interface object (e.g., a system control indicator) and if the gaze input satisfies attention criteria with respect to the first user interface object (e.g., is maintained long enough). Conditionally displaying a set of affordances for accessing system functions of the computer system based on whether the user is sufficiently paying attention to the first user interface object used to trigger display of the plurality of affordances reduces the number of inputs needed to access (e.g., provides a shortcut to) the system functions without otherwise cluttering the three-dimensional environment with visual controls (e.g., by displaying a single less intrusive element that provides the user with visual feedback about where to look to bring up the plurality of affordances, without continuously displaying the plurality of affordances).
In some embodiments, if an event for a notification satisfies timing criteria (e.g., occurred recently, within a threshold amount of time), the computer system displays content associated with the notification in response to a gaze input directed to the first user interface object. If an event for a notification does not satisfy the timing criteria, in response to a gaze input directed to the first user interface object, the computer system displays the plurality of affordances for accessing system functions without displaying the content associated with the notification. Conditionally displaying content of a notification about a triggering event based on whether the event occurred recently enough allows the user to control whether the notification is displayed, based on whether the user gazes at the first user interface object, and causes the computer system to automatically and timely display information and/or controls that are more likely to be relevant to a current context of the computer system.
In some embodiments, if the computer system has received a request to join a communication session, and the request remains active, in response to a gaze input directed to the first user interface object, the computer system displays a user interface that includes an affordance for joining the communication session. If no request to join a communication session is active, in response to a gaze input directed to the first user interface object, the computer system displays the plurality of affordances for accessing system functions without displaying a user interface with an affordance for joining a communication session. Conditionally displaying the user interface that includes an affordance for joining a communication session that is actively being requested allows the user to control whether to view the request, based on whether the user gazes at the first user interface object, and causes the computer system to automatically and timely display information and/or controls that are likely to be more relevant to a current context of the computer system (e.g., by allowing the user to respond more quickly to an active request or by simply displaying the system controls).
The processes described below enhance the operability of the devices and make the user-device interfaces more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs needed to perform an operation, providing additional control options without cluttering the user interface with additional displayed controls, performing an operation when a set of conditions has been met without requiring further user input, improving privacy and/or security, and/or additional techniques. These techniques also reduce power usage and improve battery life of the device by enabling the user to use the device more quickly and efficiently.
In addition, in methods described herein where one or more steps are contingent upon one or more conditions having been met, it should be understood that the described method can be repeated in multiple repetitions so that over the course of the repetitions all of the conditions upon which steps in the method are contingent have been met in different repetitions of the method. For example, if a method requires performing a first step if a condition is satisfied, and a second step if the condition is not satisfied, then a person of ordinary skill would appreciate that the claimed steps are repeated until the condition has been both satisfied and not satisfied, in no particular order. Thus, a method described with one or more steps that are contingent upon one or more conditions having been met could be rewritten as a method that is repeated until each of the conditions described in the method has been met. This, however, is not required of system or computer readable medium claims where the system or computer readable medium contains instructions for performing the contingent operations based on the satisfaction of the corresponding one or more conditions and thus is capable of determining whether the contingency has or has not been satisfied without explicitly repeating steps of a method until all of the conditions upon which steps in the method are contingent have been met. A person having ordinary skill in the art would also understand that, similar to a method with contingent steps, a system or computer readable storage medium can repeat the steps of a method as many times as are needed to ensure that all of the contingent steps have been performed.
In some embodiments, as shown in
When describing a XR experience, various terms are used to differentially refer to several related but distinct environments that the user may sense and/or with which a user may interact (e.g., with inputs detected by a computer system 101 generating the XR experience that cause the computer system generating the XR experience to generate audio, visual, and/or tactile feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms:
Physical environment: A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic systems. Physical environments, such as a physical park, include physical articles, such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.
Extended reality: In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In XR, a subset of a person’s physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. For example, a XR system may detect a person’s head turning and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), adjustments to characteristic(s) of virtual object(s) in a XR environment may be made in response to representations of physical motions (e.g., vocal commands). A person may sense and/or interact with a XR object using any one of their senses, including sight, sound, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, which selectively incorporates ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person may sense and/or interact only with audio objects.
Examples of XR include virtual reality and mixed reality.
Virtual reality: A virtual reality (VR) environment refers to a simulated environment that is designed to be based entirely on computer-generated sensory inputs for one or more senses. A VR environment comprises a plurality of virtual objects with which a person may sense and/or interact. For example, computer-generated imagery of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with virtual objects in the VR environment through a simulation of the person’s presence within the computer-generated environment, and/or through a simulation of a subset of the person’s physical movements within the computer-generated environment.
Mixed reality: In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory inputs, a mixed reality (MR) environment refers to a simulated environment that is designed to incorporate sensory inputs from the physical environment, or a representation thereof, in addition to including computer-generated sensory inputs (e.g., virtual objects). On a virtuality continuum, a mixed reality environment is anywhere between, but not including, a wholly physical environment at one end and virtual reality environment at the other end. In some MR environments, computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Also, some electronic systems for presenting an MR environment may track location and/or orientation with respect to the physical environment to enable virtual objects to interact with real objects (that is, physical articles from the physical environment or representations thereof). For example, a system may account for movements so that a virtual tree appears stationary with respect to the physical ground.
Examples of mixed realities include augmented reality and augmented virtuality.
Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment, or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, so that a person, using the system, perceives the virtual objects superimposed over the physical environment. Alternatively, a system may have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system composites the images or video with virtual objects, and presents the composition on the opaque display. A person, using the system, indirectly views the physical environment by way of the images or video of the physical environment, and perceives the virtual objects superimposed over the physical environment. As used herein, a video of the physical environment shown on an opaque display is called “pass-through video,” meaning a system uses one or more image sensor(s) to capture images of the physical environment, and uses those images in presenting the AR environment on the opaque display. Further alternatively, a system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, a system may transform one or more sensor images to impose a select perspective (e.g., viewpoint) different than the perspective captured by the imaging sensors. As another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof, such that the modified portion may be representative but not photorealistic versions of the originally captured images. As a further example, a representation of a physical environment may be transformed by graphically eliminating or obfuscating portions thereof.
Augmented virtuality: An augmented virtuality (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but people with faces photorealistically reproduced from images taken of physical people. As another example, a virtual object may adopt a shape or color of a physical article imaged by one or more imaging sensors. As a further example, a virtual object may adopt shadows consistent with the position of the sun in the physical environment.
Viewpoint-locked virtual object: A virtual object is viewpoint-locked when a computer system displays the virtual object at the same location and/or position in the viewpoint of the user, even as the viewpoint of the user shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the viewpoint of the user is locked to the forward facing direction of the user’s head (e.g., the viewpoint of the user is at least a portion of the field-of-view of the user when the user is looking straight ahead); thus, the viewpoint of the user remains fixed even as the user’s gaze is shifted, without moving the user’s head. In embodiments where the computer system has a display generation component (e.g., a display screen) that can be repositioned with respect to the user’s head, the viewpoint of the user is the augmented reality view that is being presented to the user on a display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the viewpoint of the user, when the viewpoint of the user is in a first orientation (e.g., with the user’s head facing north), continues to be displayed in the upper left corner of the viewpoint of the user, even as the viewpoint of the user changes to a second orientation (e.g., with the user’s head facing west). In other words, the location and/or position at which the viewpoint-locked virtual object is displayed in the viewpoint of the user is independent of the user’s position and/or orientation in the physical environment. In embodiments in which the computer system is a head-mounted device, the viewpoint of the user is locked to the orientation of the user’s head, such that the virtual object is also referred to as a “head-locked virtual object.”
Environment-locked virtual object: A virtual object is environment-locked (alternatively, “world-locked”) when a computer system displays the virtual object at a location and/or position in the viewpoint of the user that is based on (e.g., selected in reference to and/or anchored to) a location and/or object in the three-dimensional environment (e.g., a physical environment or a virtual environment). As the viewpoint of the user shifts, the location and/or object in the environment relative to the viewpoint of the user changes, which results in the environment-locked virtual object being displayed at a different location and/or position in the viewpoint of the user. For example, an environment-locked virtual object that is locked onto a tree that is immediately in front of a user is displayed at the center of the viewpoint of the user. When the viewpoint of the user shifts to the right (e.g., the user’s head is turned to the right) so that the tree is now left-of-center in the viewpoint of the user (e.g., the tree’s position in the viewpoint of the user shifts), the environment-locked virtual object that is locked onto the tree is displayed left-of-center in the viewpoint of the user. In other words, the location and/or position at which the environment-locked virtual object is displayed in the viewpoint of the user is dependent on the position and/or orientation of the location and/or object in the environment onto which the virtual object is locked. In some embodiments, the computer system uses a stationary frame of reference (e.g., a coordinate system that is anchored to a fixed location and/or object in the physical environment) in order to determine the position at which to display an environment-locked virtual object in the viewpoint of the user. An environment-locked virtual object can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object) or can be locked to a moveable part of the environment (e.g., a vehicle, animal, person, or even a representation of portion of the users body that moves independently of a viewpoint of the user, such as a user’s hand, wrist, arm, or foot) so that the virtual object is moved as the viewpoint or the portion of the environment moves to maintain a fixed relationship between the virtual object and the portion of the environment.
In some embodiments a virtual object that is environment-locked or viewpoint-locked exhibits lazy follow behavior which reduces or delays motion of the environment-locked or viewpoint-locked virtual object relative to movement of a point of reference which the virtual object is following. In some embodiments, when exhibiting lazy follow behavior the computer system intentionally delays movement of the virtual object when detecting movement of a point of reference (e.g., a portion of the environment, the viewpoint, or a point that is fixed relative to the viewpoint, such as a point that is between 5-300 cm from the viewpoint) which the virtual object is following. For example, when the point of reference (e.g., the portion of the environment or the viewpoint) moves with a first speed, the virtual object is moved by the device to remain locked to the point of reference but moves with a second speed that is slower than the first speed (e.g., until the point of reference stops moving or slows down, at which point the virtual object starts to catch up to the point of reference). In some embodiments, when a virtual object exhibits lazy follow behavior the device ignores small amounts of movment of the point of reference (e.g., ignoring movement of the point of reference that is below a threshold amount of movement such as movement by 0-5 degrees or movement by 0-50 cm). For example, when the point of reference (e.g., the portion of the environment or the viewpoint to which the virtual object is locked) moves by a first amount, a distance between the point of reference and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the point of reference to which the virtual object is locked) and when the point of reference (e.g., the portion of the environment or the viewpoint to which the virtual object is locked) moves by a second amount that is greater than the first amount, a distance between the point of reference and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the point of reference to which the virtual object is locked) and then decreases as the amount of movement of the point of reference increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintian a fixed or substantially fixed position relative to the point of reference. In some embodiments the virtual object maintaining a substantially fixed position relative to the point of reference includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the point of reference in one or more dimensions (e.g., up/down, left/right, and/or forward/backward relative to the position of the point of reference).
Hardware: There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays, vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person’s eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head-mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person’s eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person’s retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate a XR experience for the user. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and/or hardware. The controller 110 is described in greater detail below with respect to
In some embodiments, the display generation component 120 is configured to provide the XR experience (e.g., at least a visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and/or hardware. The display generation component 120 is described in greater detail below with respect to
According to some embodiments, the display generation component 120 provides a XR experience to the user while the user is virtually and/or physically present within the scene 105.
In some embodiments, the display generation component is worn on a part of the user’s body (e.g., on his/her head, on his/her hand, etc.). As such, the display generation component 120 includes one or more XR displays provided to display the XR content. For example, in various embodiments, the display generation component 120 encloses the field-of-view of the user. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device with a display directed towards the field-of-view of the user and a camera directed towards the scene 105. In some embodiments, the handheld device is optionally placed within an enclosure that is worn on the head of the user. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a XR chamber, enclosure, or room configured to present XR content in which the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) could be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions that happen in a space in front of a handheld or tripod mounted device could similarly be implemented with an HMD where the interactions happen in a space in front of the HMD and the responses of the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld or tripod mounted device relative to the physical environment (e.g., the scene 105 or a part of the user’s body (e.g., the user’s eye(s), head, or hand)) could similarly be implemented with an HMD where the movement is caused by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user’s body (e.g., the user’s eye(s), head, or hand)).
While pertinent features of the operating environment 100 are shown in
In some embodiments, the one or more communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I/O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and/or the like.
The memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some embodiments, the memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices remotely located from the one or more processing units 202. The memory 220 comprises a non-transitory computer readable storage medium. In some embodiments, the memory 220 or the non-transitory computer readable storage medium of the memory 220 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 230 and a XR experience module 240.
The operating system 230 includes instructions for handling various basic system services and for performing hardware dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for respective groups of one or more users). To that end, in various embodiments, the XR experience module 240 includes a data obtaining unit 241, a tracking unit 242, a coordination unit 246, and a data transmitting unit 248.
In some embodiments, the data obtaining unit 241 is configured to obtain data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 of
In some embodiments, the tracking unit 242 is configured to map the scene 105 and to track the position/location of at least the display generation component 120 with respect to the scene 105 of
In some embodiments, the coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by the display generation component 120, and optionally, by one or more of the output devices 155 and/or peripheral devices 195. To that end, in various embodiments, the coordination unit 246 includes instructions and/or logic therefor, and heuristics and metadata therefor.
In some embodiments, the data transmitting unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally, to one or more of the input devices 125, output devices 155, sensors 190, and/or peripheral devices 195. To that end, in various embodiments, the data transmitting unit 248 includes instructions and/or logic therefor, and heuristics and metadata therefor.
Although the data obtaining unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmitting unit 248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data obtaining unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmitting unit 248 may be located in separate computing devices.
Moreover,
In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like.
In some embodiments, the one or more XR displays 312 are configured to provide the XR experience to the user. In some embodiments, the one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some embodiments, the one or more XR displays 312 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes a XR display for each eye of the user. In some embodiments, the one or more XR displays 312 are capable of presenting MR and VR content. In some embodiments, the one or more XR displays 312 are capable of presenting MR or VR content.
In some embodiments, the one or more image sensors 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user (and may be referred to as an eye-tracking camera). In some embodiments, the one or more image sensors 314 are configured to obtain image data that corresponds to at least a portion of the user’s hand(s) and optionally arm(s) of the user (and may be referred to as a hand-tracking camera). In some embodiments, the one or more image sensors 314 are configured to be forward-facing so as to obtain image data that corresponds to the scene as would be viewed by the user if the display generation component 120 (e.g., HMD) was not present (and may be referred to as a scene camera). The one or more optional image sensors 314 can include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and/or the like.
The memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, the memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices remotely located from the one or more processing units 302. The memory 320 comprises a non-transitory computer readable storage medium. In some embodiments, the memory 320 or the non-transitory computer readable storage medium of the memory 320 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 330 and a XR presentation module 340.
The operating system 330 includes instructions for handling various basic system services and for performing hardware dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to the user via the one or more XR displays 312. To that end, in various embodiments, the XR presentation module 340 includes a data obtaining unit 342, a XR presenting unit 344, a XR map generating unit 346, and a data transmitting unit 348.
In some embodiments, the data obtaining unit 342 is configured to obtain data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of
In some embodiments, the XR presenting unit 344 is configured to present XR content via the one or more XR displays 312. To that end, in various embodiments, the XR presenting unit 344 includes instructions and/or logic therefor, and heuristics and metadata therefor.
In some embodiments, the XR map generating unit 346 is configured to generate a XR map (e.g., a 3D map of the mixed reality scene or a map of the physical environment into which computer-generated objects can be placed to generate the extended reality) based on media content data. To that end, in various embodiments, the XR map generating unit 346 includes instructions and/or logic therefor, and heuristics and metadata therefor.
In some embodiments, the data transmitting unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input devices 125, output devices 155, sensors 190, and/or peripheral devices 195. To that end, in various embodiments, the data transmitting unit 348 includes instructions and/or logic therefor, and heuristics and metadata therefor.
Although the data obtaining unit 342, the XR presenting unit 344, the XR map generating unit 346, and the data transmitting unit 348 are shown as residing on a single device (e.g., the display generation component 120 of
Moreover,
In some embodiments, the hand tracking device 140 includes image sensors 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and/or color cameras, etc.) that capture three-dimensional scene information that includes at least a hand 406 of a human user. The image sensors 404 capture the hand images with sufficient resolution to enable the fingers and their respective positions to be distinguished. The image sensors 404 typically capture images of other parts of the user’s body, as well, or possibly all of the body, and may have either zoom capabilities or a dedicated sensor with enhanced magnification to capture images of the hand with the desired resolution. In some embodiments, the image sensors 404 also capture 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensors 404 are used in conjunction with other image sensors to capture the physical environment of the scene 105, or serve as the image sensors that capture the physical environments of the scene 105. In some embodiments, the image sensors 404 are positioned relative to the user or the user’s environment in a way that a field of view of the image sensors or a portion thereof is used to define an interaction space in which hand movement captured by the image sensors are treated as inputs to the controller 110.
In some embodiments, the image sensors 404 output a sequence of frames containing 3D map data (and possibly color image data, as well) to the controller 110, which extracts high-level information from the map data. This high-level information is typically provided via an Application Program Interface (API) to an application running on the controller, which drives the display generation component 120 accordingly. For example, the user may interact with software running on the controller 110 by moving his hand 406 and changing his hand posture.
In some embodiments, the image sensors 404 project a pattern of spots onto a scene containing the hand 406 and capture an image of the projected pattern. In some embodiments, the controller 110 computes the 3D coordinates of points in the scene (including points on the surface of the user’s hand) by triangulation, based on transverse shifts of the spots in the pattern. This approach is advantageous in that it does not require the user to hold or wear any sort of beacon, sensor, or other marker. It gives the depth coordinates of points in the scene relative to a predetermined reference plane, at a certain distance from the image sensors 404. In the present disclosure, the image sensors 404 are assumed to define an orthogonal set of x, y, z axes, so that depth coordinates of points in the scene correspond to z components measured by the image sensors. Alternatively, the image sensors 404 (e.g., a hand tracking device) may use other methods of 3D mapping, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.
In some embodiments, the hand tracking device 140 captures and processes a temporal sequence of depth maps containing the user’s hand, while the user moves his hand (e.g., whole hand or one or more fingers). Software running on a processor in the image sensors 404 and/or the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in a database 408, based on a prior learning process, in order to estimate the pose of the hand in each frame. The pose typically includes 3D locations of the user’s hand joints and finger tips.
The software may also analyze the trajectory of the hands and/or fingers over multiple frames in the sequence in order to identify gestures. The pose estimation functions described herein may be interleaved with motion tracking functions, so that patch-based pose estimation is performed only once in every two (or more) frames, while tracking is used to find changes in the pose that occur over the remaining frames. The pose, motion, and gesture information are provided via the above-mentioned API to an application program running on the controller 110. This program may, for example, move and modify images presented on the display generation component 120, or perform other functions, in response to the pose and/or gesture information.
In some embodiments, a gesture includes an air gesture. An air gesture is a gesture that is detected without the user touching (or independently of) an input element that is part of a device (e.g., computer system 101, one or more input device 125, and/or hand tracking device 140) and is based on detected motion of a portion (e.g., the head, one or more arms, one or more hands, one or more fingers, and/or one or more legs) of the user’s body through the air including motion of the user’s body relative to an absolute reference (e.g., an angle of the user’s arm relative to the ground or a distance of the user’s hand relative to the ground), relative to another portion of the user’s body (e.g., movement of a hand of the user relative to a shoulder of the user, movement of one hand of the user relative to another hand of the user, and/or movement of a finger of the user relative to another finger or portion of a hand of the user), and/or absolute motion of a portion of the user’s body (e.g., a tap gesture that includes movement of a hand in a predetermined pose by a predetermined amount and/or speed, or a shake gesture that includes a predetermined speed or amount of rotation of a portion of the user’s body).
In some embodiments, input gestures used in the various examples and embodiments described herein include air gestures performed by movement of the user’s finger(s) relative to other finger(s) or part(s) of the user’s hand) for interacting with an XR environment (e.g., a virtual or mixed-reality environment), in accordance with some embodiments. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independently of an input element that is a part of the device) and is based on detected motion of a portion of the user’s body through the air including motion of the user’s body relative to an absolute reference (e.g., an angle of the user’s arm relative to the ground or a distance of the user’s hand relative to the ground), relative to another portion of the user’s body (e.g., movement of a hand of the user relative to a shoulder of the user, movement of one hand of the user relative to another hand of the user, and/or movement of a finger of the user relative to another finger or portion of a hand of the user), and/or absolute motion of a portion of the user’s body (e.g., a tap gesture that includes movement of a hand in a predetermined pose by a predetermined amount and/or speed, or a shake gesture that includes a predetermined speed or amount of rotation of a portion of the user’s body).
In some embodiments in which the input gesture is an air gesture (e.g., in the absence of physical contact with an input device that provides the computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touchscreen, or contact with a mouse or trackpad to move a cursor to the user interface element), the gesture takes into account the user’s attention (e.g., gaze) to determine the target of the user input (e.g., for direct inputs, as described below). Thus, in implementations involving air gestures, the input gesture is, for example, detected attention (e.g., gaze) toward the user interface element in combination (e.g., concurrent) with movement of a user’s finger(s) and/or hands to perform a pinch and/or tap input, as described in more detail below.
In some embodiments, input gestures that are directed to a user interface object are performed directly or indirectly with reference to a user interface object. For example, a user input is performed directly on the user interface object in accordance with performing the input gesture with the user’s hand at a position that corresponds to the position of the user interface object in the three-dimensional environment (e.g., as determined based on a current viewpoint of the user). In some embodiments, the input gesture is performed indirectly on the user interface object in accordance with the user performing the input gesture while a position of the user’s hand is not at the position that corresponds to the position of the user interface object in the three-dimensional environment while detecting the user’s attention (e.g., gaze) on the user interface object. For example, for direct input gesture, the user is enabled to direct the user’s input to the user interface object by initiating the gesture at, or near, a position corresponding to the displayed position of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0-5 cm, as measured from an outer edge of the option or a center portion of the option). For an indirect input gesture, the user is enabled to direct the user’s input to the user interface object by paying attention to the user interface object (e.g., by gazing at the user interface object) and, while paying attention to the option, the user initiates the input gesture (e.g., at any position that is detectable by the computer system) (e.g., at a position that does not correspond to the displayed position of the user interface object).
In some embodiments, input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch inputs and tap inputs, for interacting with a virtual or mixed-reality environment, in accordance with some embodiments. For example, the pinch inputs and tap inputs described below are performed as air gestures.
In some embodiments, a pinch input is part of an air gesture that includes one or more of: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture includes movement of two or more fingers of a hand to make contact with one another, that is, optionally, followed by an immediate (e.g., within 0-1 seconds) break in contact from each other. A long pinch gesture that is an air gesture includes movement of two or more fingers of a hand to make contact with one another for at least a threshold amount of time (e.g., at least 1 second), before detecting a break in contact with one another. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., with the two or more fingers making contact), and the long pinch gesture continues until a break in contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture comprises two (e.g., or more) pinch inputs (e.g., performed by the same hand) detected in immediate (e.g., within a respective time period) succession of each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between the two or more fingers), and performs a second pinch input within a respective time period (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.
In some embodiments, a pinch and drag gesture that is an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., followed by) a drag input that changes a position of the user’s hand from a first position (e.g., a start position of the drag) to a second position (e.g., an end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to make contact with one another and moves the same hand to the second position in the air with the drag gesture). In some embodiments, the pinch input is performed by a first hand of the user and the drag input is performed by the second hand of the user (e.g., the user’s second hand moves from the first position to the second position in the air while the user continues the pinch input with the user’s first hand. In some embodiments, an input gesture that is an air gesture includes inputs (e.g., pinch and/or tap inputs) performed using both of the user’s two hands. For example, the input gesture includes two (e.g., or more) pinch inputs performed in conjunction with (e.g., concurrently with, or within a respective time period of) each other. For example, a first pinch gesture performed using a first hand of the user (e.g., a pinch input, a long pinch input, or a pinch and drag input), and, in conjunction with performing the pinch input using the first hand, performing a second pinch input using the other hand (e.g., the second hand of the user’s two hands). In some embodiments, movement between the user’s two hands (e.g., to increase and/or decrease a distance or relative orientation between the user’s two hands)
In some embodiments, a tap input (e.g., directed to a user interface element) performed as an air gesture includes movement of a user’s finger(s) toward the user interface element, movement of the user’s hand toward the user interface element optionally with the user’s finger(s) extended toward the user interface element, a downward motion of a user’s finger (e.g., mimicking a mouse click motion or a tap on a touchscreen), or other movement of the user’s hand. In some embodiments a tap input that is performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of a finger or hand away from the viewpoint of the user and/or toward an object that is the target of the tap input followed by an end of the movement. In some embodiments the end of the movement is detected based on a change in movement characteristics of the finger or hand performing the tap gesture (e.g., an end of movement away from the viewpoint of the user and/or toward the object that is the target of the tap input, a reversal of direction of movement of the finger or hand, and/or a reversal of a direction of acceleration of movement of the finger or hand).
In some embodiments, attention of a user is determined to be directed to a portion of the three-dimensional environment based on detection of gaze directed to the portion of the three-dimensional environment (optionally, without requiring other conditions). In some embodiments, attention of a user is determined to be directed to a portion of the three-dimensional environment based on detection of gaze directed to the portion of the three-dimensional environment with one or more additional conditions such as requiring that gaze is directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and/or requiring that the gaze is directed to the portion of the three-dimensional environment while the viewpoint of the user is within a distance threshold from the portion of the three-dimensional environment in order for the device to determine that attention of the user is directed to the portion of the three-dimensional environment, where if one of the additional conditions is not met, the device determines that attention is not directed to the portion of the three-dimensional environment toward which gaze is directed (e.g., until the one or more additional conditions are met).
In some embodiments, the detection of a ready state configuration of a user or a portion of a user is detected by the computer system. Detection of a ready state configuration of a hand is used by a computer system as an indication that the user is likely preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., a pinch, tap, pinch and drag, double pinch, long pinch, or other air gesture described herein). For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with a thumb and one or more fingers extended and spaced apart ready to make a pinch or grab gesture or a pre-tap with one or more fingers extended and palm facing away from the user), based on whether the hand is in a predetermined position relative to a viewpoint of the user (e.g., below the user’s head and above the user’s waist and extended out from the body by at least 15, 20, 25, 30, or 50 cm), and/or based on whether the hand has moved in a particular manner (e.g., moved toward a region in front of the user above the user’s waist and below the user’s head or moved away from the user’s body or leg). In some embodiments, the ready state is used to determine whether interactive elements of the user interface respond to attention (e.g., gaze) inputs.
In scenarios where inputs are described with reference to air gestures, it should be understood that similar gestures could be detected using a hardware input device that is attached to or held by one or more hands of a user, where the position of the hardware input device in space can be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and/or one or more inertial measurement units and the position and/or movement of the hardware input device is used in place of the position and/or movement of the one or more hands in the corresponding air gesture(s). In scenarios where inputs are described with reference to air gestures, it should be understood that similar gestures could be detected using a hardware input device that is attached to or held by one or more hands of a user, user inputs can be detected with controls contained in the hardware input device such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger coverings that can detect a position or change in position of portions of a hand and/or fingers relative to each other, relative to the user’s body, and/or relative to a physical environment of the user, and/or other hardware input device controls, wherein the user inputs with the controls contained in the hardware input device are used in place of hand and/or finger gestures such as air taps or air pinches in the corresponding air gesture(s). For example, a selection input that is described as being performed with an air tap or air pinch input could be alternatively detected with a button press, a tap on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input. As another example, a movement input that is described as being performed with an air pinch and drag could be alternatively detected based on an interaction with the hardware input control such as a button press and hold, a touch on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input that is followed by movement of the hardware input device (e.g., along with the hand with which the hardware input device is associated) through space. Similarly, a two-handed input that includes movement of the hands relative to each other could be performed with one air gesture and one hardware input device in the hand that is not performing the air gesture, two hardware input devices held in different hands, or two air gestures performed by different hands using various combinations of air gestures and/or the inputs detected by one or more hardware input devices that are described above.
In some embodiments, the software may be downloaded to the controller 110 in electronic form, over a network, for example, or it may alternatively be provided on tangible, non-transitory media, such as optical, magnetic, or electronic memory media. In some embodiments, the database 408 is likewise stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in
In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) for displaying frames including left and right images in front of a user’s eyes to thus provide 3D virtual views to the user. For example, a head-mounted display generation component may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user’s eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user’s environment for display. In some embodiments, a head-mounted display generation component may have a transparent or semi-transparent display through which a user may view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, display generation component projects virtual objects into the physical environment. The virtual objects may be projected, for example, on a physical surface or as a holograph, so that an individual, using the system, observes the virtual objects superimposed over the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be necessary.
As shown in
In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine parameters of the eye tracking device for the specific operating environment 100, for example the 3D geometric relationship and parameters of the LEDs, cameras, hot mirrors (if present), eye lenses, and display screen. The device-specific calibration process may be performed at the factory or another facility prior to delivery of the AR/VR equipment to the end user. The device- specific calibration process may be an automated calibration process or a manual calibration process. A user-specific calibration process may include an estimation of a specific user’s eye parameters, for example the pupil location, fovea location, optical axis, visual axis, eye spacing, etc. Once the device-specific and user- specific parameters are determined for the eye tracking device 130, images captured by the eye tracking cameras can be processed using a glint-assisted method to determine the current visual axis and point of gaze of the user with respect to the display, in accordance with some embodiments.
As shown in
In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking cameras 540 for various purposes, for example in processing the frames 562 for display. The controller 110 optionally estimates the user’s point of gaze on the display 510 based on the gaze tracking input 542 obtained from the eye tracking cameras 540 using the glint-assisted methods or other suitable methods. The point of gaze estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.
The following describes several possible use cases for the user’s current gaze direction, and is not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user’s gaze. For example, the controller 110 may generate virtual content at a higher resolution in a foveal region determined from the user’s current gaze direction than in peripheral regions. As another example, the controller may position or move virtual content in the view based at least in part on the user’s current gaze direction. As another example, the controller may display particular virtual content in the view based at least in part on the user’s current gaze direction. As another example use case in AR applications, the controller 110 may direct external cameras for capturing the physical environments of the XR experience to focus in the determined direction. The autofocus mechanism of the external cameras may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another example use case, the eye lenses 520 may be focusable lenses, and the gaze tracking information is used by the controller to adjust the focus of the eye lenses 520 so that the virtual object that the user is currently looking at has the proper vergence to match the convergence of the user’s eyes 592. The controller 110 may leverage the gaze tracking information to direct the eye lenses 520 to adjust focus so that close objects that the user is looking at appear at the right distance.
In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens(es) 520), eye tracking cameras (e.g., eye tracking camera(s) 540), and light sources (e.g., light sources 530 (e.g., IR or NIR LEDs), mounted in a wearable housing. The light sources emit light (e.g., IR or NIR light) towards the user’s eye(s) 592. In some embodiments, the light sources may be arranged in rings or circles around each of the lenses as shown in
In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and thus does not introduce noise in the gaze tracking system. Note that the location and angle of eye tracking camera(s) 540 is given by way of example, and is not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user’s face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user’s face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user’s face. In some embodiments, a camera 540 that operates at one wavelength (e.g., 850 nm) and a camera 540 that operates at a different wavelength (e.g., 940 nm) may be used on each side of the user’s face.
Embodiments of the gaze tracking system as illustrated in
As shown in
At 610, for the current captured images, if the tracking state is YES, then the method proceeds to element 640. At 610, if the tracking state is NO, then as indicated at 620 the images are analyzed to detect the user’s pupils and glints in the images. At 630, if the pupils and glints are successfully detected, then the method proceeds to element 640. Otherwise, the method returns to element 610 to process next images of the user’s eyes.
At 640, if proceeding from element 610, the current frames are analyzed to track the pupils and glints based in part on prior information from the previous frames. At 640, if proceeding from element 630, the tracking state is initialized based on the detected pupils and glints in the current frames. Results of processing at element 640 are checked to verify that the results of tracking or detection can be trusted. For example, results may be checked to determine if the pupil and a sufficient number of glints to perform gaze estimation are successfully tracked or detected in the current frames. At 650, if the results cannot be trusted, then the tracking state is set to NO at element 660, and the method returns to element 610 to process next images of the user’s eyes. At 650, if the results are trusted, then the method proceeds to element 670. At 670, the tracking state is set to YES (if not already YES), and the pupil and glint information is passed to element 680 to estimate the user’s point of gaze.
In the present disclosure, various input methods are described with respect to interactions with a computer system. When an example is provided using one input device or input method and another example is provided using another input device or input method, it is to be understood that each example may be compatible with and optionally utilizes the input device or input method described with respect to another example. Similarly, various output methods are described with respect to interactions with a computer system. When an example is provided using one output device or output method and another example is provided using another output device or output method, it is to be understood that each example may be compatible with and optionally utilizes the output device or output method described with respect to another example. Similarly, various methods are described with respect to interactions with a virtual environment or a mixed reality environment through a computer system. When an example is provided using interactions with a virtual environment and another example is provided using mixed reality environment, it is to be understood that each example may be compatible with and optionally utilizes the methods described with respect to another example. As such, the present disclosure discloses embodiments that are combinations of the features of multiple examples, without exhaustively listing all features of an embodiment in the description of each example embodiment.
Attention is now directed towards embodiments of user interfaces (“UI”) and associated processes that may be implemented on a computer system, such as a portable multifunction device or a head-mounted device, in communication with a display generation component, one or more input devices, and optionally one or cameras.
In some embodiments, a three-dimensional environment that is visible via a viewport provided by a display generation component described herein is a virtual three-dimensional environment that includes virtual objects and content at different virtual positions in the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at different virtual positions in the three-dimensional environment that are constrained by one or more physical aspects of the physical environment (e.g., positions and orientations of walls, floors, surfaces, direction of gravity, time of day, and/or spatial relationships between physical objects). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. In some embodiments, the representation of the physical environment includes respective representations of physical objects and surfaces at different positions in the three-dimensional environment, such that the spatial relationships between the different physical objects and surfaces in the physical environment are reflected by the spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. In some embodiments, when virtual objects are placed relative to the positions of the representations of physical objects and surfaces in the three-dimensional environment, they appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment. In some embodiments, the computer system transitions between displaying the different types of environment (e.g., transitions between presenting a computer-generated environment or experience with different levels of immersion, and/or adjusting the relative prominence of audio/visual sensory inputs from the virtual content and from the representation of the physical environment) based on user inputs and/or contextual conditions.
In some embodiments, the display generation component includes a pass-through portion in which the representation of the physical environment is displayed. In some embodiments, the pass-through portion of the display generation component is a transparent or semi-transparent (e.g., see-through) portion of the display generation component revealing at least a portion of physical environment surrounding and within the field of view of a user. For example, the pass-through portion is a portion of a head-mounted display or heads-up display that is made semi-transparent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% of opacity) or transparent, such that the user can see through it to view the real world surrounding the user without removing the head-mounted display or moving away from the heads-up display. In some embodiments, the pass-through portion gradually transitions from semi-transparent or transparent to fully opaque when displaying a virtual or mixed reality environment. In some embodiments, the pass-through portion of the display generation component displays a live feed of images or video of at least a portion of physical environment captured by one or more cameras (e.g., rear facing camera(s) of a mobile device or associated with a head-mounted display, or other cameras that feed image data to the computer system). In some embodiments, the one or more cameras point at a portion of the physical environment that is directly in front of the user’s eyes (e.g., behind the display generation component relative to the user of the display generation component). In some embodiments, the one or more cameras point at a portion of the physical environment that is not directly in front of the user’s eyes (e.g., in a different physical environment, or to the side or behind the user).
In some embodiments, when displaying virtual objects at positions that correspond to locations of one or more physical objects in the physical environment (e.g., at positions in a virtual reality environment, a mixed reality environment, or an augmented reality environment), at least some of the virtual objects are displayed in place of (e.g., replacing display of) a portion of the live view (e.g., a portion of the physical environment captured in the live view) of the cameras. In some embodiments, at least some of the virtual objects and content are projected onto physical surfaces or empty space in the physical environment and are visible through the pass-through portion of the display generation component (e.g., viewable as part of the camera view of the physical environment or through the transparent or semi-transparent portion of the display generation component). In some embodiments, at least some of the virtual objects and virtual content are displayed to overlay a portion of the display and blocks the view of at least a portion of the physical environment visible through the transparent or semi-transparent portion of the display generation component.
In some embodiments, the display generation component displays different views of the three-dimensional environment in accordance with user inputs or movements that change the virtual position of the viewpoint of the currently displayed view of the three-dimensional environment relative to the three-dimensional environment. In some embodiments, when the three-dimensional environment is a virtual environment, the viewpoint moves in accordance with navigation or locomotion requests (e.g., in-air hand gestures, and/or gestures performed by movement of one portion of the hand relative to another portion of the hand) without requiring movement of the user’s head, torso, and/or the display generation component in the physical environment. In some embodiments, movement of the user’s head and/or torso, and/or the movement of the display generation component or other location sensing elements of the computer system (e.g., due to the user holding the display generation component or wearing the HMD), relative to the physical environment cause corresponding movement of the viewpoint (e.g., with corresponding movement direction, movement distance, movement speed, and/or change in orientation) relative to the three-dimensional environment, resulting in corresponding change in the currently displayed view of the three-dimensional environment. In some embodiments, when a virtual object has a respective spatial relationship relative to the viewpoint (e.g., is anchored or fixed to the viewpoint, and/or maintains its spatial relationship to the viewport through which the three-dimensional environment is visible), movement of the viewpoint relative to the three-dimensional environment would cause movement of the virtual object relative to the three-dimensional environment while the position of the virtual object in the field of view and/or viewport is maintained (e.g., the virtual object is said to be head locked). In some embodiments, a virtual object is body-locked to the user, and moves relative to the three-dimensional environment when the user moves as a whole in the physical environment (e.g., carrying or wearing the display generation component and/or other location sensing component of the computer system), but will not move in the three-dimensional environment in response to the user’s head movement alone (e.g., the display generation component and/or other location sensing component of the computer system rotating around a fixed location of the user in the physical environment). In some embodiments, a virtual object is, optionally, locked to another portion of the user, such as a user’s hand or a user’s wrist, and moves in the three-dimensional environment in accordance with movement of the portion of the user in the physical environment, to maintain a spatial relationship between the position of the virtual object and the virtual position of the portion of the user in the three-dimensional environment. In some embodiments, a virtual object is locked to a portion of a field of view provided by the display generation component (e.g., has a fixed spatial relationship to a viewport through which the three-dimensional environment is visible, and/or has a fixed spatial relationship to the viewpoint of the user), and moves in the three-dimensional environment in accordance with the movement of the field of view inside the viewport, irrespective of movement of the user that does not cause a change of the field of view inside the viewport.
In some embodiments, as shown in
In some embodiments, the indicator 7010 of system function menu is displayed in a periphery region (e.g., within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of an edge) of a viewport provided by the first display generation component through which the three-dimensional environment is visible (e.g., relative to a field of view of a user using the first display generation component). As shown in
In some embodiments, the indicator 7010 of system function menu continues to be displayed, even if the user interacts with other user interfaces, user interface objects, and/or user interface elements in the three-dimensional environment. For example, the indicator 7010 of system function menu continues to be displayed even if the user’s attention is directed at a virtual object 7012. The user can interact with or manipulate the virtual object 7012 (e.g., with the user’s gaze, air gestures, and/or verbal inputs), and the computer system 7100 will continue to display the indicator 7010 of system function menu. In some embodiments, the indicator 7010 of system function menu is visually deemphasized (e.g., dimmed and/or blurred) when the user interacts with other user interfaces, user interface objects, and/or user interface elements (e.g., to avoid distracting the user while the user’s attention (e.g., gaze) is directed to other user interface elements).
In some embodiments, the indicator 7010 of system function menu is displayed while a user’s attention satisfies proximity criteria with respect to the indicator 7010 of system function menu, and the indicator 7010 of system function menu ceases to be displayed in response to detecting that a user’s attention does not satisfy the proximity criteria with respect to the indicator 7010 of system function menu. For example, if the user’s attention moves to the center of the display generation component of the computer system 7100, the computer system 7100 ceases to display the indicator 7010 of system function menu. In some embodiments, after ceasing to display the indicator 7010 of system function menu, the indicator 7010 of system function menu is redisplayed in response to the user’s attention again satisfying the proximity criteria (e.g., in response to detecting that the user’s attention returns to a location near the indicator 7010 of system function menu (e.g., as shown in
In
In some embodiments, the indicator 7010 of system function menu changes appearance as the indicator 7010 of system function menu moves from the location 7016 to the location 7018, and/or from the location 7018 to the final position shown in
The system function menu 7024 includes a plurality of affordances for accessing system functions of the computer system 7100. For example, as shown in
In some embodiments, the system function menu 7024 is displayed within a threshold distance (e.g., within 0.5 cm, 1 cm, 2 cm, or 5 cm) of the indicator 7010 of system function menu. In some embodiments, the system function menu 7024 is displayed with a respective spatial relationship to (e.g., directly below, directly above, to the right of, or to the left of, or other required spatial relationships) the indicator 7010 of system function menu. In some embodiments, the system function menu 7024 is displayed on top of other visible user interfaces in a respective view of the three-dimensional environment. For example, if an application launching user interface and/or application user interface is displayed (e.g., prior to displaying the system function menu 7024, and at least partially overlapping the region over which system function menu 7024 would be displayed), the system function menu 7024 when invoked is displayed over at least a portion of the application launching user interface and/or application user interface. In some embodiments, the system function menu 7024 is displayed so as to appear closer to a viewpoint of the user than the application launching user interface and/or application user interface (e.g., because the system function menu 7024 is displayed over at least a portion of the application launching user interface and/or application user interface, and/or because the system function menu 7024 appears larger than the application launching user interface and/or application user interface).
In some embodiments, displaying the system function menu 7024 includes displaying an animated transition (e.g., of the system function menu 7024 appearing). For example, the animated transition may include an animation of the system function menu 7024 expanding downward from the indicator 7010 of system function menu. Alternately, or in addition, the animated transition may include an animation of the system function menu 7024 gradually appearing or fading into view. In some embodiments, the animated transition that is displayed depends on a position of the indicator 7010 of system function menu. For example, if the indicator 7010 of system function menu is instead displayed near the left edge of the display generation component, the animated transition optionally includes an animation of the system function menu 7024 expanding outward from the indicator 7010 of system function menu toward the right. More generally, the animated transition optionally includes an animation of the system function menu 7024 expanding out from the indicator 7010 of system function menu (e.g., toward a center of the view of the three-dimensional environment that is visible). In some embodiments, the indicator 7010 of system function menu is visually deemphasized (e.g., dimmed, faded, grayed out, and/or blurred) or not displayed while the system function menu 7024 is displayed, as indicated in
In some embodiments, as shown in
In some embodiments, the computer system 7100 ceases to display the system function menu 7024 when (e.g., in response to detecting that) the user’s gaze no longer satisfies the attention criteria (e.g., the user’s attention 7116 is no longer directed to the indicator 7010 of system function menu or the system function menu 7024). In some embodiments, the system function menu 7024 may remain displayed for a time (e.g., a number of seconds, optionally accounting for lazy follow behavior so as to allow the system function menu 7024 to finish settling into a new viewpoint-locked position) even after the user’s gaze no longer satisfies the attention criteria, and may remain displayed if the user’s gaze subsequently satisfies the attention criteria again (e.g., within a threshold amount of time of ceasing to satisfy the attention criteria). For example, in embodiments where the indicator 7010 of system function menu exhibits lazy follow behavior, the system function menu 7024 may remain displayed even if the user’s gaze does not continuously remain directed to the indicator 7010 of system function menu (e.g., the user’s gaze does not precisely track the indicator 7010 of system function menu) as the viewpoint of the user moves (e.g., as the user moves a touchscreen device or turns their head while wearing a head-mounted display).
In some embodiments, in response to detecting that the user’s attention 7116 is directed to the volume affordance 7038, the computer system 7100 outputs additional content associated with the volume affordance 7038 (e.g., a description of a volume setting associated with the volume affordance 7038, instructions for adjusting the volume setting associated with the volume affordance 7038, and/or a current value of the volume setting associated with the volume affordance 7038), which is sometimes referred to as a “tool tip.” In some embodiments, the tool tip is displayed after a delay (e.g., of 1 second, 5 seconds, or any threshold time between 0 and 10 seconds) in response to detecting that the user’s attention 7116 is directed to the volume affordance 7038. In some embodiments, the tool tip is displayed adjacent to (e.g., immediately above, immediately below, to the left of, or to the right of) the volume affordance 7038. In some embodiments, the tool tip is displayed partially overlaid over at least a portion of the volume affordance 7038 (and/or other affordances in the system function menu 7024).
In some embodiments, the system space 7040 remains displayed while the user’s attention is directed to the system space 7040, the volume affordance 7038, or the indicator 7010 of system function menu. In some embodiments, the system space 7040 ceases to be displayed in response to the user’s attention 7116 moving away from the volume affordance 7038 or the system space 7040. In some embodiments, the system space 7040 ceases to be displayed a threshold amount of time (e.g., 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, or 60 seconds) after the user’s attention has moved away from the volume affordance 7038 or the system space 7040 (e.g., to avoid ceasing to display the system space 7040 if the user did not intend to dismiss the system space 7040, for example if the user were briefly distracted, or to account for hardware limitations (e.g., a camera of the computer system 7100 temporarily losing focus of the user due to movement of the camera and/or changes in environmental lighting), for less than the threshold amount of time).
In some embodiments, if the system space 7040 is already displayed (e.g., in response to detecting an activation input directed to the volume affordance 7038), and the computer system 7100 detects that the user’s attention shifts to another affordance (e.g., the control affordance 7046 or any other affordance in system function menu 7024), the computer system 7100 ceases to display the system space 7040 and displays a new system space for the other affordance (and optionally replaces display of the system space 7040 with the system space for the other affordance). In some embodiments, the system space for the other affordance is displayed in response to another activation input while the user’s attention is directed to the other affordance.
In some embodiments, while the system space 7040 is displayed, in response to detecting the user’s attention directed to the system function menu 7024, the system space 7040 is visually deemphasized (e.g., dimmed, faded, and/or blurred). While the system space 7040 is displayed, in response to detecting the user’s attention directed to the system space 7040, the system function menu 7024 is visually deemphasized (and, optionally, the system space 7040 ceases to be visually deemphasized). In some embodiments, the indicator 7010 of system function menu is visually deemphasized (e.g., dimmed, faded, and/or blurred), or not displayed, when the user 7002 is gazing at or interacting with system function menu 7024 or system space 7040.
In some scenarios, the computer system 7100 displays an application launching user interface (e.g., a home user interface, optionally including one or more affordances corresponding respectively to one or more applications that can be launched on computer system 7100). In some embodiments, the indicator 7010 of system function menu is displayed while the application launching user interface is displayed, and the application launching user interface remains displayed while system function menu 7024 is displayed in response to user interaction with indicator 7010 of system function menu as described herein (e.g., prior to displaying the system space 7040). In some such embodiments, in response to detecting the activation input directed to the volume affordance 7038, as described herein with reference to
In some scenarios, the computer system 7100 displays an application user interface prior to displaying the system space 7040 (e.g., concurrently with indicator 7010 of system function menu and/or system function menu 7024). In response to detecting the activation input directed to the volume affordance 7038, the computer system 7100 displays the system space 7040 overlaid on at least a portion of the application user interface.
As shown in
In some embodiments, the system space 7040 includes a plurality of sliders for adjusting volumes for the computer system 7100. For example, the system space 7040 includes a first slider for adjusting a volume associated with application (e.g., the volume for content or media within an application user interface, or the volume of notifications associated with applications), a second slider for adjusting a volume associated with people/contacts (e.g., the volume for audio calls, video calls, and AR/VR communication sessions with other users), and a third slider for adjusting a volume associated with environments (e.g., a volume for AR or VR experiences). In some embodiments, the computer system 7100 determines the active context (e.g., an application user interface is being displayed and/or the attention of the user is directed to an application user interface; there is an active communication session; and/or an AR or VR experience is active) and displays a single slider for adjusting the volume for the active context. In some embodiments, the computer system 7100 determines there is no active context (e.g., no application user interface is displayed, no communication sessions are active, and no AR/VR experiences are active), and displays the plurality of sliders for adjusting volumes for different contexts (e.g., the three contexts described above).
In some embodiments, the text field of the system space 7050 updates in real time (e.g., as the user 7002 is speaking). In some embodiments, the text field of the system space 7050 displays the text of the detected verbal input only after (e.g., in response to) detecting completion of the verbal input (and optionally after detecting that the user 7002 has stopped speaking for a threshold amount of time). In some embodiments, in response to detecting (e.g., completion of) the verbal input, the computer system 7100 automatically performs one or more functions (e.g., an Internet search, an application search, a document or file search, or other content search) associated with the verbal input. In some embodiments, the computer system 7100 performs the one or more functions associated with the verbal input in response to detecting a first portion of the verbal input, and continues to perform the one or more functions while the verbal input continues (e.g., the search is continually updated as additional portions of the verbal input are detected). For example, the computer system 7100 performs a search function in response to detecting the verbal input (e.g., presenting search results for the word “Weather” in
In some embodiments, for a respective affordance in system function menu 7024, no system space is displayed in response to an activation input directed to the respective affordance. For example, as shown in
In some embodiments, a system space that is displayed in response to activation of an affordance in system function menu 7024 (e.g., any of the system spaces shown in
Additional descriptions regarding
As shown in the examples in
In some scenarios, the computer system displays other user interfaces, such as a user interface 7058, in the three-dimensional environment. In some scenarios, the user interface 7058 is an application user interface. In some scenarios, the user interface 7058 is an application launching user interface (e.g., a home user interface).
In response to detecting a first gaze input directed to the user interface object 7056, as shown by the user’s attention 7116 directed to the user interface object 7056 in
In some embodiments, the notification content 7060 is displayed over a portion of the user interface 7058 (and/or over other displayed user interfaces in the view of the three-dimensional environment). In some embodiments, the notification content 7060 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of the user interface object 7056. In some embodiments, the notification content 7060 is displayed with a respective spatial relationship to (e.g., directly below, directly above, to the right of, or to the left of) the user interface object 7056. In some embodiments, the notification content 7060 is displayed closer to a viewpoint of the user than other user interface objects such as user interface 7058 (e.g., to simulate being “on top” of other user interface elements, relative to the viewpoint of the user).
In some embodiments, the notification content 7060 is displayed concurrently with the user interface object 7056. In some embodiments, as indicated by the dashed outline of user interface object 7056 in
The dashed outline of the system function menu 7024 in
In some embodiments, the computer system 7100 switches between display of the notification content 7060 and the system function menu 7024 in response to successive gaze inputs directed to the user interface object 7056. In response to detecting the first gaze input directed to the user interface object 7056, the computer system 7100 displays the notification content 7060. After displaying the notification content 7060, the user gazes away from the user interface object 7056 (and optionally, the notification content 7060 ceases to be displayed). After gazing away from the user interface object 7056, the user’s gaze returns to the user interface object 7056, and in response, the computer system 7100 displays the system function menu 7024 (e.g., without displaying the notification content 7060). If the user repeats this process (e.g., looks away from the user interface object 7056, then looks back at the user interface object 7056), the computer system 7100 displays (e.g., redisplays) the notification content 7060 (e.g., without displaying the system function menu 7024). This allows the user to switch between displaying the notification content 7060 and the system function menu 7024 without cluttering the displayed user interface (e.g., occupying too much of the user’s field of view with concurrent display of the notification content 7060 and the system function menu 7024).
In some embodiments, the computer system 7100 switches between display of the notification content 7060 and additional notification content (e.g., for a second event for a second notification that also satisfies the timing criteria). In some such embodiments, the computer 7100 optionally switches between display of the notification content 7060, the additional notification content, and the system function menu 7024 (e.g., cycles through these different user interfaces) in response to successive gaze inputs to the user interface object 7056 (e.g., the displayed user interface may change each time the user looks away from, then back at, the user interface object 7056).
In some embodiments, the user’s attention 7116 is directed to the notification content 7060, as shown in
In some embodiments, the user interface object 7056, the notification content 7060, the system function menu 7024, and/or the application user interface 7062 follow the viewpoint of the user 7002 as the viewpoint of the user 7002 moves (e.g., as the user moves a touchscreen device or turns their head while wearing a head-mounted display) (e.g., one or more are viewpoint-locked). In some embodiments, the follow behavior of the user interface object 7056, the notification content 7060, the system function menu 7024, and/or the application user interface 7062 is lazy follow behavior. In some embodiments, the user interface object 7056 has lazy follow behavior. In some embodiments, the follow behavior of the user interface object 7056 is the same as the lazy follow behavior of the indicator 7010 of system function menu as described above with reference to
In some embodiments, the user interface object 7056 has different follow (e.g., lazy follow) behavior than the system function menu 7024 (e.g., similar to how the indicator 7010 of system function menu optionally has different follow behavior from the system function menu 7024, as described above, and as shown in
In some embodiments, the computer system 7100 ceases to display the notification content 7060 when (e.g., in response to detecting that) the user’s attention 7116 is no longer directed to the user interface object 7056 (or the notification content 7060). In some embodiments, the notification content 7060 may remain displayed for a respective period of time (e.g., a number of seconds, optionally accounting for lazy follow behavior) even after the user’s attention 7116 is no longer directed to the user interface object 7056 (or notification content 7060), and remains displayed if the user’s attention 7116 is subsequently directed back to the user interface object 7056 (or the notification content 7060) (e.g., the respective time period is restarted when the user’s attention 7116 returns to the user interface object 7056 before the expiration of the respective time period). In some such embodiments, the notification content 7060 ceases to be displayed if the computer system 7100 detects that the user’s attention 7116 is no longer directed to the user interface object 7056 (or the notification content 7060), and has not returned to the user interface object 7056 (or the notification content 7060) within the respective period of time. In some embodiments, after the notification content 7060 ceases to be displayed, in response to the user’s attention 7116 subsequently being directed to the user interface object 7056 (e.g., while the first event for the first notification satisfies the timing criteria), the computer system 7100 redisplays the notification content 7060.
For example, in embodiments where the user interface object 7056 (and/or notification content 7060) exhibit(s) lazy follow behavior, the notification content 7060 may remain displayed even if the user’s attention 7116 does not continuously remain directed to the user interface object 7056 (or the notification content 7060). This is advantageous, for example, so that the user’s attention 7116 does not need to precisely track the user interface object 7056 (or the notification content 7060) while the user (and/or the viewpoint of the user) is moving in order for the notification content 7060 to remain displayed, thus preventing situations where the notification content 7060 ceases to be displayed due to the user’s attention 7116 having moved away for too long and the user must perform another gaze input directed to the user interface object 7056 in order to redisplay the notification content 7060.
After the first event for the first notification no longer satisfies the timing criteria (e.g., the notification is no longer considered recent enough), the user interface object 7056 returns to a default appearance (e.g., as shown by the indicator 7010 of system function menu in
In some embodiments, the user interface object 7056 returns to the default appearance (e.g., appears instead as the indicator 7010 of system function menu in
Additional descriptions regarding
As shown in the examples in
In some embodiments, the user interface object 7064 is displayed if a request for the computer system 7100 to join a communication session satisfies first timing criteria (e.g., the communication session request has been active for less than a threshold amount of time), the user interface object 7056 is displayed if a first event for a first notification satisfies second timing criteria (e.g., different from the first timing criteria) (e.g., timing criteria as described herein with reference to
In some embodiments, the user interface object 7064 has an appearance that includes an indication of an application associated with the request for the computer system 7100 to join the communication session (e.g., the user interface object 7064 appears as the application icon for an application associated with the communication session). In some such embodiments, the user interface object 7064 optionally includes different indications for different types of communication sessions (e.g., a phone call, a video call, and/or a shared virtual experience). In some embodiments, the user interface object 7064 has an appearance that indicates a user associated with the request (e.g., the user that initiated the request) for the computer system 7100 to join the communication session (e.g., as a name, initials, a username, an avatar, a photograph, and/or the like of the initiating user).
In some embodiments, in accordance with a determination that a request for the computer system 7100 to join a communication session satisfies timing criteria, the user interface object 7064 has a different color (e.g., is green, or another color) than when the request to join the communication session no longer satisfies the timing criteria. In some embodiments, the user interface object 7064 has a different shape (e.g., as compared to the indicator 7010 of system function menu) in addition to, or instead of, having the different color. In some embodiments, in accordance with a determination that a request for the computer system 7100 to join a communication session satisfies timing criteria, the user interface object 7064 displays an animation (e.g., is animated). For example, the user interface object 7064 bounces up and down, the border of the user interface object 7064 pulses, the user interface object 7064 changes in size, and/or the user interface object 7064 rotates. In some embodiments, the animation is displayed in combination with an audio output (e.g., a beep, a series of tones (e.g., that is optionally repeated), and/or a ring tone). In some embodiments, the animation is displayed without any corresponding audio output.
If the user’s attention 7116 is directed to the user interface object 7064 while a request for the computer system 7100 to join a communication session satisfies timing criteria (e.g., as indicated by timer 7066-2 (
In some embodiments, the user interface 7068 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of the user interface object 7064. In some embodiments, the user interface 7068 is displayed with a respective spatial relationship to (e.g., directly below, directly above, to the right of, to the left of, or other required spatial relationships) the user interface object 7064. In some embodiments, the user interface 7068 is displayed over at least a portion of other visible user interfaces in a respective view of the three-dimensional environment (e.g., as shown in
The dashed outline of user interface object 7064 in
In some embodiments, displaying the user interface 7068 includes displaying an animated transition (e.g., of the user interface 7068 appearing). For example, the animated transition may include an animation of the user interface 7068 expanding downward from the user interface object 7064. Alternately, or in addition, the animated transition may include an animation of the user interface 7068 gradually getting bigger and/or fading into view. In some embodiments, the animated transition depends on a position of the user interface object 7064. For example, if the user interface object 7064 is instead displayed near the left edge of the display generation component, the animated transition optionally includes an animation of the user interface 7068 expanding outward from the user interface object 7064 towards the right edge of the display generation component.
In some embodiments, the user interface object 7064, the system function menu 7024, and/or the user interface 7068 follow the viewpoint of the user 7002 as the viewpoint of the user 7002 moves (e.g., one or more are viewpoint-locked). In some embodiments, the follow behavior of the user interface object 7064, the system function menu 7024, and/or the user interface 7068 is lazy follow behavior. In some embodiments, the user interface object 7064, the system function menu 7024, and/or the user interface 7068 exhibit distinctive follow (e.g., lazy follow) behavior from each other (e.g., follow a viewpoint of the user more or less closely than another user interface element).
In some embodiments, after displaying the user interface 7068, the computer system 7100 ceases to display the user interface 7068 in response to the user 7002 dismissing user interface 7068 (e.g., by no longer gazing at the user interface object 7064 or the user interface 7068) or after displaying the user interface 7068 for a threshold amount of time. After the user interface 7068 ceases to be displayed, the user interface object 7064 optionally returns to a default appearance (e.g., the appearance of the indicator 7010 of system function menu), indicating that the user has viewed the request to join the communication session (e.g.,
In some embodiments, if the user’s attention 7116 is not directed to the user interface object 7064 before the request for the computer system 7100 to join the communication session expires (e.g., before the initiating user cancels the communication session request) and/or no longer satisfies the timing criteria (e.g., more than a threshold amount of time TH1 has elapsed since the request was first received), the computer system 7100 forgoes displaying the user interface 7068 for joining the communication session (e.g., even in response to the user’s attention 7116 being directed to the user interface object 7064). As shown in
For example, if the communication session request for the computer system 7100 lasts for 1 minute, and the user’s attention 7116 is directed to the user interface object 7064 within that 1 minute, then the computer system 7100 displays the user interface 7068 for joining the communication session (e.g., as shown in
In the example shown in
The call control user interface 7072 includes a plurality of affordances for accessing functions of the active communication session. For example, the call control user interface 7072 includes any combination of the following: a messaging affordance (e.g., for launching a separate application (e.g., a text messaging application) for communicating with one or more other users in the communication session and/or viewing previous communications (e.g., conversation transcripts) with one or more other users in the communication session; an information affordance for displaying additional information about the communication session (e.g., a list of users in the communication session, and/or an indication of the duration of the communication session); a microphone affordance (e.g., for muting or unmuting the user 7002 in the communication session); a session history affordance (e.g., for displaying shared content that was previously shared between one or more users in the communication session); a sharing affordance (e.g., for sharing content with other users in the communication session and/or displaying one or more shared content items that were previously shared (e.g., with the other users in the communication session)); and/or an exit affordance (e.g., labeled “Leave,” for removing the user from the communication session and/or for ending the communication session for all users).
In some embodiments, the call control user interface 7072 is concurrently displayed with the system function menu 7024, and/or with the user interface object 7064 (e.g., as shown in
In some embodiments, the display characteristics (e.g., displayed location, spatial relationships to other user interface elements, displayed animation(s), and/or follow or lazy follow behavior), of the call control interface 7072 are analogous to those of the user interface 7068 and/or the user interface 7070 described above.
In some embodiments, the call control user interface 7072 includes an affordance for switching between different representations of one or more users in the communication session (e.g., a profile picture, a two-dimensional image of the user, the user’s name, and/or the user’s initials). In some embodiments, the affordance is used to switch between displaying a realistic likeness of the user (e.g., which may be two- or three-dimensional, including for example an image of the user, a video or video stream of the user, and/or an avatar that looks like the user’s and/or is animated based on the user’s movements) and an abstract likeness or abstract representation of the user (e.g., the user’s name, the user’s initials, and/or an image that is not based on the user’s likeness). For example, if a respective user’s avatar is displayed when the affordance for switching between the realistic likeness and abstract likeness is activated, the respective user’s avatar is replaced by the abstract likeness. If the abstract likeness is displayed when the affordance for switching between the realistic likeness and abstract likeness is activated, the abstract likeness is replaced by the respective user’s avatar. In some embodiments, the call control user interface 7072 includes an affordance for switching between different visual representations of one or more users in the communication session (e.g., switching between one or more of an avatar, a three-dimensional character, an icon, a profile picture, a user’s name, and a user’s initials).
In some embodiments, the computer system 7100 ceases to display the call control user interface 7072 when (e.g., in response to detecting that) the user’s attention 7116 is no longer directed to the user interface object 7064 (or the call control user interface 7072). In some embodiments, the call control user interface 7072 remains displayed for a respective period of time (e.g., 1 second, 5 seconds, or a time period accounting for lazy follow behavior) even after the user’s attention 7116 is no longer directed to the user interface object 7064 (or the call control user interface 7072), and remains displayed if the user’s attention 7116 is subsequently directed back to the user interface object 7064 (or the call control user interface 7072) (e.g., the respective time period is restarted when the user’s attention 7116 returns to the user interface object 7064 before the expiration of the respective time period. In some such embodiments, the call control user interface 7072 ceases to be displayed if the computer system 7100 detects that the user’s attention 7116 is no longer directed to the user interface object 7064 (or the call control user interface 7072), and has not returned to the user interface object 7064 (or the user interface 7072) within the respective period of time, and the user interface object 7064 remains displayed, or is redisplayed if the user interface object 7064 was not displayed when the computer system 7100 ceased to display the call control user interface 7072 (e.g., so that the call control user interface 7072 can be redisplayed when the user’s attention 7116 is subsequently directed to the user interface object 7072). In some embodiments, after the call control user interface 7072 ceases to be displayed, if the user’s attention 7116 subsequently is directed to the user interface object 7064 while the computer system 7100 is still in the communication session, the computer system 7100 redisplays the call control user interface 7072 in response. While this behavior is described above with reference to the call control user interface 7072, this behavior optionally also pertains to the user interface 7068 and/or user interface 7070, which may be displayed concurrently with, or instead of, the call control user interface 7072.
In some embodiments, the user interface object 7064, system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and/or the call control user interface 7072 are head-locked virtual objects (e.g., viewpoint-locked virtual objects as described herein). In some embodiments, when the viewpoint of the user changes (e.g., the user turns and/or moves to a new position in the three-dimensional environment, such that the view of the three-dimensional environment changes), the user interface object 7064 moves in tandem with the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and/or the call control user interface 7072, if displayed (e.g., such that the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and/or the call control user interface, if displayed, maintain respective required spatial relationships with the user interface object 7064, as described herein). In some embodiments, the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and/or the call control user interface 7072 move in tandem with each other (e.g., if the system function menu 7024 and the call control user interface 7072 are concurrently displayed, the system function menu 7024 moves in tandem with the call control user interface 7072, such that the call control user interface 7072 maintains a respective spatial relationship with the system function menu 7024 (e.g., in an analogous manner to maintaining a respective spatial relationship with the user interface object 7064)).
In some embodiments, after the request for the computer system 7100 to join the communication session no longer satisfies the timing criteria (e.g., the request to join the communication session expires or times out), the user interface object 7064 returns to a default appearance (e.g., as shown by the indicator 7010 of system function menu in
In some embodiments, the computer system 7100 switches between display of the user interface 7068, the system function menu 7024, and optionally, one or more additional user interface elements (e.g., the notification content 7060 described above with reference to
In an example scenario in which both a request to join a communication session satisfies first timing criteria (e.g., was received less than a time TH1 ago, as described herein with reference to
In some embodiments, the computer system 7100 switches between display of the user interface 7068 and another request to join a communication session (e.g., a second request to join a second communication session that also satisfies the timing criteria). In some such embodiments, the computer 7100 optionally switches between display of the user interface 7068, and a second user interface for joining the second communication session (e.g., an analogous but distinct user interface that includes a representation of a different user who initiated the request for the second communication session), and the system function menu 7024 (e.g., the displayed user interface element may change each time the user looks away from, then back at, the user interface object 7064).
For telephone communication sessions (e.g., telephone calls), in step 7074, the computer system 7100 (e.g., initially) detects an incoming request to join the telephone communication session. Before a first time threshold THC1 since first receiving the telephone communication session request (e.g., while a request to join a telephone call is ringing), the request to join the telephone communication session satisfies timing criteria. As shown in step 7076, in response to detecting the incoming request to join the telephone communication session, the computer system 7100 displays a request to join the telephone communication session (e.g., incoming call user interface 7068 (
After time THC1, the request to join the telephone communication session no longer meets the timing criteria (e.g., the request has timed out). At step 7078, the current time is after THC1, but is before a second time threshold THC2. If the user gazes at the user interface object 7064 between time threshold THC1 and time threshold THC2, as in step 7078, the computer system 7100 displays in response a missed call user interface (e.g., the user interface 7070 (
If the user gazes at the user interface object 7064 as in step 7082, because the current time is after both THC1 and THC2, the request to join the telephone communication session does not satisfy the timing criteria (e.g., the amount of time since receiving the request is not less than THC1), and the gaze input is not detected within the threshold amount of time since receiving the request (e.g., the amount of time since receiving the request is not less than THC2). Thus, in response to detecting the gaze input of step 7082, and as shown in step 7084, the computer system 7100 does not display an incoming call user interface nor a missed call user interface, and instead displays the system function menu 7024 (e.g., that includes a plurality of affordances for performing system operations associated with the computer system 7100, as described herein for example with reference to
For video communication sessions (e.g., video calls), in step 7086, the computer system 7100 (e.g., initially) detects an incoming request to join the video communication session. Before a first time threshold THV1 since first receiving the video communication session request (e.g., while a request to join a video call is ringing), the request to join the video communication session satisfies timing criteria. If, as shown in step 7088, the user’s gaze is directed to the user interface object 7064 before time threshold THV1, then, in response to detecting the user’s gaze directed to the user interface object 7064 (e.g., and because the current time is before THV1), the computer system 7100 displays an incoming call user interface (e.g., the user interface 7068 (
After time THV1, the request to join the video communication session no longer meets the timing criteria (e.g., the request has timed out). At step 7092, the current time is after THV1, but is before a second time threshold THV2. If the user gazes at the user interface object 7064 between time threshold THV1 and time threshold THV2, as in step 7092, the computer system 7100 displays in response a missed call user interface (e.g., the user interface 7070 (
If the user gazes at the user interface object 7064 as in step 7096, because the current time is after both THV1 and THV2, the request to join the video communication session does not satisfy the timing criteria (e.g., the amount of time since receiving the request is not less than THV1), and the gaze input is not detected within the threshold amount of time since receiving the request (e.g., the amount of time since receiving the request is not less than THV2). Thus, in response to detecting the gaze input of step 7096, and as shown in step 7098, the computer system 7100 does not display an incoming call user interface nor a missed call user interface, and instead displays the system function menu 7024. As used herein, “video communication session” or “video call” refers to videotelephony protocols or to telephony protocols for which video is supported yet can be enabled or disabled by the user (e.g., within the same telephony service, the user may choose between videotelephony and audio-only telephony, or turn off their own and/or another user’s video). As used herein, “telephone communication session” or “telephone call” refers to audio-only telephony (e.g., including some cellular and Internet telephony) for which video is not supported. One of ordinary skill will readily appreciate that audio-only calls over videotelephony protocols may instead be considered “telephone calls” rather than “video calls.”
For extended reality (XR) communication sessions, in step 7102, the computer system 7100 detects an incoming request to join the XR communication session. Before a time threshold THx since first receiving the XR communication session request (e.g., while the requesting user keeps the invitation to join the XR communication session open), the request to join the XR communication session satisfies timing criteria. If, as shown in step 7104, the user’s gaze is directed to the system control indicator (e.g., the user interface object 7064 (
After time THx, the request to join the XR communication session no longer meets the timing criteria (e.g., the request has timed out). If the user gazes at the user interface object 7064 after time threshold THx, as in step 7108, the computer system 7100 displays the system function menu 7024, as in step 7110. In the example shown in
In some embodiments, video communication requests are more persistent than telephone communication requests, in that an incoming video call rings longer than an incoming phone call. This is shown visually in
Additional descriptions regarding
In some embodiments (e.g., as described above with reference to
In
In some embodiments, the system function menu 7024 includes a plurality of affordances for accessing system functions of the computer system 7100. For example, as shown in
In
In some embodiments, user selection of a respective one of the affordances displayed in the system function menu 7024 (e.g., the home affordance 7124, the search affordance 7042, the volume affordance 7038, the notification affordance 7044, the control affordance 7046, the virtual assistant affordance 7048 and/or any other affordance) causes the computer system 101 to display a corresponding user interface for the selected affordance. For example, the user interface 7136 is a home user interface (e.g., as described in greater detail below, with reference to
In some embodiments, the user interface 7136 is a search user interface for performing a search operation (e.g., if the user’s attention is directed to the search affordance 7042, rather than the control affordance 7046). In some embodiments, the user interface 7136 includes a search bar that provides visual feedback regarding the search term. In some embodiments, the user 7002 inputs a search term with a verbal input (e.g., and at least one input device of the one or more input devices of the computer system 7100 is a microphone configured to detect verbal inputs). In some embodiments, the user 7002 enters the search term by gazing at the search bar displayed in the user interface 7136 (e.g., the computer system 7100 detects the user’s attention directed to the search bar, and determines that the user 7002 intends to perform a search function), and/or performs a verbal input (e.g., speaking aloud the desired search term). In some embodiments, the search bar updates in real time, as the user performs the verbal input (e.g., each spoken word of the search term is displayed in the search bar as the user finishes speaking the word).
In some embodiments, the user interface 7136 includes a virtual keyboard (e.g., optionally in addition to the search bar), and the user 7002 manually inputs a search term, in the search bar, using the virtual keyboard. In some embodiments, the user 7002 enters the search term either through a verbal input, or through the virtual keyboard. In some embodiments, the user 7002 enters the search term through a combination of input devices (e.g., the user begins entering the search term through a verbal input, and edits and/or completes the search term through the virtual keyboard, or vice-versa).
In some embodiments, the user interface 7136 is a user interface for adjusting a volume setting of the computer system 7100 (e.g., in response to detecting that the user’s gaze is directed to the volume affordance 7038, rather than the control affordance 7046). In some embodiments, the user interface 7136 includes one or more controls (e.g., a volume slider, a dial, and/or one or more buttons) for adjusting volume settings of the computer system 7100.
In some embodiments, the user interface 7136 is a notification user interface for displaying content from one or more notifications received by the computer system 7100 (e.g., in response to detecting that the user’s attention is directed to the notification affordance 7044, rather than the control affordance 7046).
In some embodiments, the user interface 7136 is a system user interface for accessing additional system settings of the computer system 7100 that are not included in the system function menu 7024. For example, the affordances 7138, 7140, 7142, and/or 7144 of the user interface 7136 allow the user to access and/or adjust different settings for the computer system 7100 (e.g., optionally corresponding to settings that are not accessible or adjustable directly from the system function menu 7024).
In
In
In some embodiments, one or more virtual objects (e.g., that are optionally locked, or anchored, to the three-dimensional environment) are displayed with a simulated size that corresponds to a distance between where the virtual object is located in the three-dimensional environment relative to the user’s current position. For example, the virtual object 7012 and the user interface 7136 are both environment-locked (e.g., also referred to herein as anchored to a position in the three-dimensional environment), and as the user moves to the new location 7026-d, the virtual object 7012 and the user interface 7136 are displayed with a smaller size (e.g., as compared to their respective sizes in
In some embodiments, the system function menu 7024 is not environment-locked, and is instead head-locked, such that the system function menu 7024 has the same size (e.g., as compared to the size of the system function menu 7024 in
In
In some embodiments, the affordances 7148, 7150, 7152, and 7154 correspond to respective applications of the computer system 7100 (e.g., the affordances are application icons and/or other representations of applications), and the user interacts with a respective affordance corresponding to a respective application to launch an application user interface for the respective application (e.g., by opening the application and/or displaying a new application window for the application). In some embodiments, the affordances 7148, 7150, 7152, and 7154 correspond to respective contacts for respective other users (e.g., the affordances are user avatars, contact information, telephone numbers, user IDs, or entity names), stored in the memory of the computer system 7100, and the user 7002 interacts with a respective affordance corresponding to a respective contact to initiate a communication session with the contact (e.g., another user associated with another computer system distinct from computer system 7100). In some embodiments, the affordances 7148, 7150, 7152, and 7154 correspond to respective virtual environments (e.g., different AR environments, different VR environments, different AR experiences, and/or different VR experiences), and the user 7002 interacts with a respective affordance corresponding to a respective virtual environment to initiate display of the respective virtual environment in the three-dimensional environment.
In some embodiments, the user interface 7146 includes different groups of affordances (e.g., a first group of affordances for launching application user interfaces, a second group of affordances for initiating communication session with other users, and/or a third group of affordances for initiating display of different virtual environments). In some embodiments, different groups of affordances are displayed in the user interface 7146 in different contexts (e.g., not every affordance is always displayed, and/or not every group of affordances is always displayed). In some embodiments, the user 7002 switches between different groups of affordances (e.g., by performing an air gesture (e.g., an air tap or an air pinch), by interacting with a specific affordance for switching between groups of affordances, and/or by interacting with a respective affordance for displaying a respective group of affordances). In some embodiments, the user interface 7146 includes additional affordances (e.g., in addition to, or instead of, the affordances 7148, 7150, 7152, and 7154), and the user 7002 scrolls between display of the additional affordances in the user interface 7146 (e.g., by performing an air gesture, such as an air tap or an air pinch). In some embodiments, the user 7002 performs a first type of air gesture (e.g., a tap gesture) to switch between different groups of affordances, and the user 7002 performs a second type of air gesture (e.g., a pinch and drag gesture) to scroll display of affordances (e.g., optionally within a displayed group of affordances).
In some embodiments, affordances of the user interface 7146 that are within a threshold distance from (e.g., proximate to, near, and/or next to) a specific boundary (or boundaries) of the user interface 7146 are displayed with different visual characteristics (e.g., are displayed with a degree of fading or blurring or other visual deemphasis) than affordances that are outside of the threshold distance. In some embodiments, the specific boundary (or boundaries) of the user interface 7146 near which affordances are displayed with different visual characteristics is associated with a direction in which display of the affordances of the user interface 7146 can be scrolled. For example, if the user 7002 can scroll display of the affordances of the user interface 7146 in an upward and/or downward directions, the affordances (e.g., affordances 7150 and 7154) near the lower boundary (and/or the upper boundary) of the user interface 7146 are displayed with a degree of fading or blurring. In some embodiments, the user 7002 can scroll display of the affordances of the user interface 7146 in a leftward (and/or rightward direction), and affordances (e.g., affordances 7152 and 7154) near the right boundary (and/or the left boundary) of the user interface 7146 are displayed with a degree of fading or blurring. In some embodiments, affordances that are within the threshold distance from any of a plurality of boundaries (e.g., an upper and lower boundary, a left and right boundary, an upper and right boundary, a lower and left boundary, an upper and left boundary, or a lower and right boundary, or another combination of boundaries, of the user interface 7146) are displayed with the different visual characteristics (e.g., visual deemphasis). In some embodiments, the boundary (or boundaries) are not straight edges (e.g., the user interface 7146 is circular in shape, and affordances near the outer boundary of the circle are displayed with the different visual characteristics, while affordances closer to the center of the user interface 7146 are not displayed with the different visual characteristics). In some embodiments, optionally only a portion, less than all, of an affordance that is proximate to the specific boundary of the user interface 7146 is displayed with the different visual characteristics (e.g., a lower portion of the affordances 7150 and 7154 is displayed as faded or blurred, while an upper portion of the affordances 7150 and 7154 is displayed without fading or blurring (e.g., with the same, or different, visual characteristics as the affordances 7148 and 7152).
In some embodiments, the visual characteristics of the affordances that are within the threshold distance from the specific boundary of the user interface 7146 vary in accordance with the distance from the specific boundary. For example, lower portions of affordances 7150 and 7154 (e.g., that are closest to the lower boundary of the user interface 7146) are displayed with a first degree (e.g., a high degree) of blurring or fading, upper portions of the affordances 7150 and 7154 are displayed with a second degree of blurring or fading (e.g., not blurred or faded at all), and portions of the affordances 7150 and 7154 between the lower and upper portions of the affordances 7150 and 7154 are displayed with a third degree of blurring or fading (optionally an intermediate level of blurring or fading between the first degree and the second degree) (e.g., less blurred or faded than the lower portions, but more blurred or faded than the upper portions, of the affordances 7150 and 7154).
In some embodiments, affordances of the user interface 7146 that are within a threshold distance from a simulated boundary (or boundaries) of the user interface 7146 are displayed with different visual characteristics (e.g., are displayed with a degree of fading or blurring) than affordances that are outside of the threshold distance from the simulated boundary. For example, if the user interface 7146 had a circular shape, the simulated boundary could be a right “edge” of the user interface 7146 (even though the user interface 7146 itself has no corresponding boundary), and affordances near the 2 o′clock, 3 o′clock, and 4 o′clock positions are displayed with the different visual characteristics (e.g., and optionally, the affordances near the 3 o′clock position are displayed with a greater change in the visual characteristic (e.g., more blurring or fading) than the affordances near the 2 o′clock and 4 o′clock positions).
In some embodiments, the visual characteristics include a simulated thickness (e.g., an affordance with a high level of simulated thickness appears more, or fully, three-dimensional, whereas an affordance with a low level of simulated thickness appears flatter or more two-dimensional).
In some embodiments, the user interface 7156 is an application user interface (e.g., the affordance 7148 is an application icon, and the user launches the application user interface 7156 by selecting the affordance 7148). In some embodiments, the user interface 7156 includes controls for a phone call or a virtual reality session, and/or is a user interface that includes video for a video call or other communication session, with another user (e.g., the affordance 7148 is a user avatar, and the user 7002 initiates a communication session with the other user via the affordance 7148). In some embodiments, the user interface 7156 is associated with (e.g., is, or is an element of) a virtual environment (e.g., the three-dimensional environment displayed on the display of the computer system 7100).
For example, like the user interface 7136 described above, the user interface 7146 (and/or the user interface 7156) is also environment-locked. In response to the user 7002′s movement to the location 7026-e, the user interface 7146 (and optionally, the user interface 7156) have a different size and position in the view of the three dimensional environment that is visible via the viewport provided by the display generation component of the computer system 7100. In
The user’s attention 7116, is directed to the indicator 7010 of system function menu, and in response to the user’s attention 7116 satisfying attention criteria (e.g., the same attention criteria described above with reference to
Additional descriptions regarding
In some embodiments, region 7158 and region 7160 are not visible (e.g., as indicated by the dotted outlines for region 7158 and region 7160 in
In some embodiments, region 7158 is larger than region 7160. In some embodiments, region 7158 includes region 7160 (e.g., region 7160 is a sub-region for region 7158). In some embodiments, region 7158 and region 7160 are viewpoint-locked regions and exhibit analogous behavior to the indicator 7010 of system function menu and/or the system function menu 7024, as described above with reference to
In some embodiments, one or both of region 7158 and region 7160 have a different shape than the substantially square shape shown in
In
In
In some embodiments, the computer system 7100 displays (e.g., concurrently with the indicator 7010 of system function menu) instructions (e.g., text-based instructions, visual or pictorial instructions) for expanding the indicator 7010 of system function menu and/or invoking the system function menu 7024 (e.g., via one or more inputs as described below with reference to
In some embodiments, the indicator 7010 of system function menu is not displayed if it would overlap with an existing user interface element. For example, if the user interface 7136 (
In some embodiments, the indicator 7010 of system function menu is viewpoint-locked (e.g., similarly to region 7158 and region 7160), and has analogous behavior as described with reference to, and as shown in,
In some embodiments, the location of the indicator 7010 of system function menu is selected at least in part based on a position of the user relative to the computer system 7100. In some embodiments, the indicator 7010 of system function menu is displayed in a consistent region (e.g., upper right corner, or another region consistently chosen by the computer system) of the display of the computer system 7100, but the exact location of the indicator 7010 of system function menu varies within the region (e.g., the exact location of the indicator 7010 of system function menu can be up to a threshold distance away from a respective default position), based at least in part on the position of the user relative to the computer system 7100. In some embodiments, the computer system 7100 may include one or more cameras (or other sensors) for tracking a position of the user, for determining the orientation (e.g., an amount of tilt and/or rotation) of the computer system 7100 relative to the user.
For example, if the computer system 7100 is a handheld device, the computer system 7100 detects that the orientation of the computer system 7100 is not exactly aligned with the user (e.g., when the user holds only the right side of the computer system 7100 (relative to the user’s view) with only the user’s right hand, a horizontal axis of the computer system 7100 (that would normally run from the left to the right of the user’s view) is slightly rotated (e.g., runs from a slightly lower left to a slightly higher right, in the user’s view) due to the weight of the computer system 7100 and the lack of support (e.g., via the user’s left hand) on the left side of the computer system 7100). The computer system 7100 displays the indicator 7010 of system function menu at an adjusted location (e.g., closer to the top edge and/or the right edge of the computer system 7100), in accordance with some embodiments.
In some embodiments, the computer system 7100 is a head-mounted display (HMD), and the computer system 7100 accommodates for how the device is worn by the user. For example, the computer system 7100 may be oriented differently for different users, due to differences in the physical characteristics of each user (e.g. different sizes and positions of each user’s ears, eyes and/or nose), asymmetry of a user’s physical features (e.g., ears that have slightly different heights, eyes that have different sizes, and/or an eyeline that is not perfectly horizontal), or other external factors (e.g., the presence of earphones/headphones, glasses, or other accessories that are worn on or around the user’s head). For example, the computer system 7100 detects that the head-mounted display sits lower on the user’s face than another user (or lower than a default or average height), and the computer system 7100 displays the indicator 7010 of system function menu at an adjusted location (e.g., closer to the top edge of the computer system 7100).
Displaying the indicator 7010 of system function menu at an adjusted location allows the indicator 7010 of system function menu to be displayed at a consistent location (e.g., relative to the user) even if the computer system 7100 is not “perfectly” aligned with the user (e.g., no tilt or rotation relative to the user), and also maintains the indicator 7010 of system function menu at a consistent location through minor movement of the computer system 7100 (e.g., due to unsteadiness of the user’s hand(s)). In some embodiments, the adjusted location of the indicator 7010 of system function menu is within a threshold distance (e.g., 0.5 mm, 1 mm, 2 mm, 5 mm, or 1 cm) of a respective location (e.g., the center point of a respective upper right portion of the display of the computer system 7100). If the adjusted location of the indicator 7010 of system function menu would require the indicator 7010 of system function menu to be displayed at a location that is greater than the threshold distance, the computer system 7100 forgoes displaying the indicator 7010 of system function menu at the adjusted location (e.g., because the adjusted location would require the indicator 7010 of system function menu to be displayed somewhere beyond the physical boundary of the display of the computer system 7100, or because the indicator 7010 of system function menu cannot be easily interacted with at the adjusted location (e.g., the indicator 7010 of system function menu would be displayed at a location that is uncomfortable for the user to interact with, and/or the indicator 7010 of system function menu would be displayed at a location where other user interface elements commonly appear (e.g., somewhere near the center portion of the display of the computer system 7100) and/or other user interface elements are already displayed. Instead of displaying the indicator 7010 of system function menu at the adjusted location is beyond the threshold distance from the respective location, the computer system 7100 instead displays a visual indication to the user (e.g., instructions for correcting the orientation and/or position of the computer system 7100 relative to the user, which would allow the indicator 7010 of system function menu to be displayed at a location that is within the threshold distance of the respective location).
In some embodiments, if the location of the indicator 7010 of system function menu is adjusted (e.g., the indicator 7010 of system function menu is displayed at the adjusted location as described above), the system function menu 7024 is also adjusted by the same amount (e.g., in the same direction(s)). In some embodiments, if the location of the indicator 7010 of system function menu is adjusted (e.g., the indicator 7010 of system function menu is displayed at the adjusted location as described above), other user interface elements (e.g., the user interface 7058 in
In
In
In some embodiments, region 7160 expands in a manner that is at least partially symmetrical with respect to the indicator 7010 of system function menu. For example, as shown in
In
In some embodiments, while the user’s attention 7116 is outside region 7158 (and/or region 7160), the indicator 7010 of system function menu shrinks at a continuous rate. In some embodiments, the indicator 7010 of system function menu shrinks at a different rate (e.g., faster than or slower than) as compared to the rate of expansion while the user’s attention 7116 is within region 7160. In some embodiments, the indicator 7010 of system function menu begins to shrink after the user’s attention 7116 is outside region 7158 (and/or region 7160) for a threshold amount of time (e.g., 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, or 60 seconds), which prevents the indicator 7010 of system function menu from shrinking if the attention of the user is temporarily redirected from region 7158 (and/or region 7160) (e.g., if the user is momentarily distracted by something in the real world or virtual environment/three-dimensional environment). While the user’s attention 7116 is outside region 7158 (and/or region 7160) and before the threshold amount of time has passed, the indicator 7010 of system function menu is maintained at a respective size (e.g., the size of the indicator 7010 of system function menu when the user’s attention 7116 moved away from region 7160). In some embodiments, if the user’s attention 7116 returns to region 7160 before the threshold amount of time has passed, the indicator 7010 of system function menu resumes expanding.
In
In
In some embodiments, the indicator 7010 of system function menu is not displayed (e.g., in any of
In some embodiments, where the indicator 7010 of system function menu is not displayed (e.g., or not subsequently displayed, after the first time the system function menu 7024 is displayed), the computer system 7100 displays (e.g., or redisplays) the system function menu 7024 when the user’s attention 7116 is directed to region 7160 and/or the location in region 7160 where the indicator 7010 of system function menu is shown in
In some embodiments, the location of the system function menu 7024 is selected at least in part based on a position of the user relative to the computer system 7100 (e.g., in an analogous fashion to the indicator 7010 of system function menu, as described above with reference to
With respect to
In some embodiments, while the system function menu 7024 is displayed, the user speaks a verbal input, and the computer system 7100 performs a function associated with the spoken verbal input. For example, the system function menu 7024 includes an affordance associated with a virtual assistant of the computer system 7100, and the user can speak a verbal input instructing the virtual assistant to perform a function of the computer system 7100 (e.g., as described in further detail above, with reference to
In some embodiments, the user’s attention 7116 can be directed to portions of the user interface other than the particular affordance for the virtual assistant or search function (e.g., the indicator 7010 of system function menu, or other portions of the system function menu 7024), and the computer system 7100 determines the appropriate function to perform based on the verbal input. For example, if the verbal input matches a list of functions of the virtual assistant, the computer system 7100 performs a function corresponding to a respective function of the list of functions corresponding to the verbal input. If the verbal input does not match any function in the list of functions of the virtual assistant, the computer system 7100 instead performs a search function based on the verbal input.
In some embodiments, the user’s attention 7116 is directed away from the system function menu 7024 (e.g., and/or the indicator 7010 of system function menu, and/or region 7160), and in response, the computer system ceases to display the system function menu 7024. In some embodiments, the computer system ceases to display the system function menu 7024 in response to detecting that the user’s attention 7116 is not directed to the system function menu 7024 (and/or the indicator 7010 of system function menu, and/or region 7160) for a threshold amount of time (e.g., 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, or 60 seconds). This allows the user’s attention to briefly wander (e.g., due to external distractions and/or eye fatigue) without accidentally ceasing to display the system function menu 7024.
In some embodiments (e.g., where the system function menu 7024 is displayed in response to detecting that the user’s attention 7116 is directed to region 7160), the computer system 7100 ceases to display the system function menu 7024 in response to detecting that the user’s attention 7116 is directed to a location outside the larger region 7158. In some embodiments, the computer system 7100 maintains display of the system function menu 7024 while the user’s attention remains directed to a location within region 7160 (and/or region 7158).
In some embodiments, after ceasing to display the system function menu 7024, the computer system detects that the user’s attention 7116 returns to the indicator 7010 of system function menu (e.g., or a location within region 7160), and in response, the computer system redisplays the system function menu 7024.
Additional descriptions regarding
In some embodiments, if the user’s attention 7116 is not directed to the indicator 7010 of system function menu (e.g., or a region such as region 7160 in
In some embodiments, the computer system 7100 displays different content (e.g., different user interfaces, or different content within the same user interface) in different contexts. In
In
In
In some embodiments, the system function menu 7024 (e.g., the default content) is at least partially displayed (e.g., concurrently with the incoming call user interface 7068). In some embodiments, if the system function menu 7024 is concurrently displayed with the incoming call user interface 7068, the system function menu 7024 is displayed with a lower level of prominence than the incoming call user interface 7068 (e.g., the system function menu 7024 is displayed dimmer than, blurrier than, more occluded or obscured than, and/or smaller than the incoming call user interface 7068). In
In some embodiments, the computer system 7100 displays additional content items, in addition to the incoming call user interface 7068 (and/or system function menu 7024). In
In some embodiments, the computer system 7100 displays a visual indication that other additional content items are available (e.g., the smaller size of the notification 7157 indicates that other notification content is available for display, but not currently displayed). In some embodiments, the visual indication that other additional content items are available is displayed with a reduced level of prominence (e.g., further from the viewpoint of the user, with a lower brightness, with a lower opacity, and/or with a higher degree of blur effect) as compared to the additional content items (e.g., the notification 7157 has a smaller size, and appears behind, the notification 7154). In some embodiments, one or more of the additional content items are displayed in place of (e.g., replace display of) the incoming call user interface 7068). In some embodiments, one or more of the additional content items are concurrently displayed with the incoming call user interface 7068.
In some embodiments, while displaying the additional content items, the computer system 7100 detects occurrence of an event (e.g., corresponding to a new notification), and the computer system 7100 displays a new notification corresponding to the event (e.g., optionally, at the location where the notification 7148 is displayed at in
In some embodiments, one or more contextual user interfaces (e.g., the incoming call user interface 7068 and/or the additional content items in
In some embodiments, the computer system 7100 displays up to a threshold number (e.g., a maximum number) of additional content items (and/or total user interface objects). For example, the threshold number is six total user interface objects, so in
In some embodiments, the user 7002 can display additional content associated with a specific contextual user interface, by directing the user’s attention 7116 to the specific contextual user interface and performing a first type of user input (e.g., a first type of air gesture, input via a hardware controller, and/or verbal input). For example, if the user’s attention 7116 is directed to the incoming call user interface 7068 and the user performs the first type of user input, the computer system 7100 displays additional information regarding the communication session corresponding to the incoming call user interface 7068 (e.g., contact information for the contact “John Smith” and/or other contacts or users who are active in the communication session), optionally in an expanded version of the incoming call user interface 7068, in accordance with some embodiments. If the user’ attention 7116 is directed to a notification 7148 and the user performs the first type of user input, the computer system 7100 displays additional notification content for the notification 7148, in accordance with some embodiments. In some embodiments, the computer system ceases to display other contextual user interfaces (e.g., other than the contextual user interface that the user’s attention is directed to) in response to detecting a user input of the first type.
In some embodiments, the computer system 7100 navigates through (e.g., scrolls display of) the displayed additional content items (e.g., to display the other additional content items) in a first direction that corresponds to where the user’s attention 7116 is detected relative to the center point/region of the display of the computer system 7116 (e.g., the first direction is leftward in
In some embodiments, as shown in
In some embodiments, the computer system 7100 navigating through the displayed additional content items in a second direction (e.g., opposite the first direction) in response to detecting that the user’s attention 7116 has moved to a different location (e.g., a location to the right of the center point/region, such as a location over the system function menu 7024, as shown in
In some embodiments, the user can perform a respective gesture (e.g., an air gesture, such as an air tap or an air pinch, or another selection input) in order to quickly return to the earlier (or earliest) displayed user interface elements. For example, in
In some embodiments, multiple context criteria are met. For example, second context criteria are met because music is playing on the computer system 7100, and simultaneously, first context criteria are met because a video call request is also active for the computer system 7100. In some embodiments, when multiple context criteria are met, the computer system 7100 determines the appropriate content to display based on a respective priority order. For example, in the respective priority order, the second context has a higher priority than the first context. When music is playing on the computer system 7100 and a video call request is also active for the computer system 7100, the computer system 7100 displays the music user interface 7166 in response to detecting the user’s attention 7116 directed to the indicator 7010 of system function menu, because the second context (e.g., a context in which music is playing) has a higher priority than the first context (e.g., a context in which the computer system is receiving an active video call request), in accordance with some embodiments.
In some embodiments, the respective priority order is determined based on a state of the device when the user’s attention 7116 is directed to the indicator 7010 of system function menu (or a user interface object that replaces the indicator 7010 of system function menu, such as the user interface object 7064 of
In some embodiments, the computer system 7100 provides a preview corresponding to contextual user interfaces that are available for display. For example, the user interface object 7064 (e.g., in
In some embodiments, the respective priority order is determined based at least in part on time (e.g., a time when a particular contextual user interface/content was generated or first displayed). In some embodiments, the contextual user interfaces are ordered in reverse temporal order (e.g., the newest contextual user interfaces have a higher priority). In some embodiments, the respective priority order determines the order of different categories of contextual user interfaces, and within each category, the contextual user interfaces of the same category are ordered by time (e.g., in reverse temporal order).
Some exemplary categories of contextual user interfaces include: (1) a notification category that includes the notifications 7148, 7150, 7152, 7154, 7156, 7158, 7160, 7162, and 7164 (of
In some embodiments, as shown in
In some embodiments, the indicator 7010 of system function menu is replaced by an appropriate user interface object (e.g., the user interface object 7064 of
While
Other possible examples of contextual user interfaces include application-related user interfaces, such as notifications, a user interface corresponding to a wireless sharing protocol (e.g., to transfer and/or share data with other computer systems), and/or a user interface displaying application function progress (e.g., a download progress for an active download, or update progress for an update of an application of the computer system 7100). In some embodiments, the computer system 7100 displays contextual user interfaces for a plurality of distinct applications (e.g., with one or more of the contextual user interfaces being displayed as additional content, for example, as shown in
In some embodiments, in response to detecting that the user’s attention 7116 is no longer directed to one of the contextual user interfaces and/or the indicator 7010 of system function menu (e.g., for a threshold amount of time, such as 1 second, 2 seconds, or 5 seconds), the computer system ceases to display the contextual user interfaces. In some embodiments, the computer system 7100 ceases to display the contextual user interfaces in response to detecting a respective air gesture (e.g., in combination with a gaze input), a touch gesture, an input provided via a controller, and/or a voice command. In some embodiments, the computer system 7100 ceases to display the contextual user interfaces in response to detecting a user input that includes movement in a respective direction. For example, if the user input continues to move towards the right of the display of the computer system 7100 in
Additional descriptions regarding
In some embodiments, some types of notifications (e.g., “system alerts” as referred to herein) do not cause the user interface object 7056 to be displayed. For example, system alerts relating to battery levels, object avoidance, and/or other users joining active communication sessions, do not cause the user interface object 7056 to be displayed (e.g., because the content of the system alert is time critical and/or high importance). In some embodiments, system alerts are displayed below the indicator 7010 of system function menu. In some embodiments, instead of displaying the user interface object 7056 (or another analogous user interface object at the same position as the user interface object 7056 in
In some embodiments, system alerts can be dismissed by directing the user’s attention to the system alert, and then directing the user’s attention away from the system alert. In some embodiments, system alerts are dismissed when the system function menu 7024 is displayed (e.g., if the user’s attention 7116 is directed to the indicator 7010 of system function menu). In some embodiments, the system alerts are concurrently displayed with the system function menu 7024 (e.g., if the user’s attention 7116 is directed to the indicator 7010 of system function menu), but the system alerts are dismissed if the user’s attention 7116 is directed to the system function menu 7024. In some embodiments, system alerts automatically disappear (e.g., are automatically dismissed) after a respective amount of time (e.g., 0.1, 0.2, 0.5, 1, 2, 5, 10, or 30 seconds, or a longer amount of time such as 1 minute, 5 minutes, or 10 minutes). In some embodiments, system alerts do not appear in the notification center (e.g., the notification center as described above with reference to
In some embodiments, the user of the computer system 7100 can access one or more functions and/or settings of the computer system 7100 via a system alert (e.g., a system alert for battery level includes an affordance for accessing the battery settings of the computer system 7100). In some embodiments, the system alert includes one or more selectable options for performing operations corresponding to the system alert (e.g., or an event corresponding to the system alert). In some embodiments, the one or more selectable options for performing operations overlaps with (e.g., are the same as, or share functionality with) the one or more functions and/or settings of the computer system 7100.
In some embodiments, the user interface 7168 is displayed at the same location as the system function menu 7024 (e.g., but the system function menu 7024 is not concurrently displayed with the user interface 7168). In some embodiments, the user interface 7168 is displayed concurrently with the system function menu 7024. In some embodiments, the user interface 7168 is displayed as partially overlapping, obscuring, or occluding the system function menu 7024 (e.g., as shown in
In some embodiments, while the user interface 7168 is displayed, the computer system 7100 displays additional content (e.g., additional notification content not visible in
For example, the user interface 7168 includes first notification content associated with a recently received notification, and the additional content includes second notification content associated with the recently received notification that is different from the first notification content. In some embodiments, the user interface 7168 includes a preview of notification content (e.g., less than all of the notification content) and the additional content includes the full notification content (e.g., an “expanded” version of the notification that includes all the notification content). In some embodiments, the additional content is (e.g., concurrently) displayed with one or more of the contextual user interfaces described above with reference to
In some embodiments, after the computer system 7100 ceases to display the user interface 7168, the user’s attention returns to the user interface object 7056 within a threshold amount of time (e.g., before the time T3), and the computer system 7100 redisplays the user interface 7168 (e.g., and/or the user interface object 7056). After the threshold amount of time (e.g., at the time T3 or later), if the user’s attention is directed to the indicator 7010 of system function menu (which is now displayed in place of the user interface object 7056), the computer system 7100 does not redisplay the user interface 7168, and optionally displays (or redisplays) the system function menu 7024 and/or any relevant contextual user interfaces (e.g., as described above with reference to
In some embodiments, after the time T3, the user can access the previously displayed user interface 7168 (e.g., and/or content of the user interface 7168) through other means (e.g., through one or more user interfaces for displaying recent notifications, or previously received notifications). For example, if the user interface 7168 was a notification, the user can access a notification history user interface of the computer system 7100, by first invoking the system function menu 7024 (e.g., by directing the user’s attention 7116 to the indicator 7010 of system function menu in
Additional descriptions regarding
In some embodiments, notification content is grouped by application. For example, the notification 7170 is a notification for a first application (e.g., Application 1), the notification 7172 is a notification for a second application (e.g., Application 2, which is different than Application 1), the group 7174 is a representative notification for a group of notifications for a third application (e.g., Application 3, which is different from Application 1 and Application 2), and the notification 7178 is a notification for a fourth application (e.g., Application 4, which is different from Application 1, Application 2, and Application 3).
In some embodiments, multiple notifications for a single application are grouped together (e.g., and represented as a “stack” of notifications). For example, an indication 7176 appears below (e.g., and behind) the group 7174 for the third application, indicating that there is at least one other notification for the third application that is not currently shown (e.g., fully shown on the display of the computer system 7100 in
In some embodiments, the notification content includes notifications for up to a maximum number of different applications. For example, in
In some embodiments, the notifications 7170, 7172, 7174, and 7178 have an appearance that is at least partially based on the physical and/or virtual environment that is displayed behind the notifications 7170, 7172, 7174, and 7178. For example, the representation 7014′ of the physical object 7014 is displayed behind the notifications 7170 and 7172 (e.g., the notifications 7170 and 7172 are displayed in front of, or on top of, the representation 7014′ in the view of the user 7002). The lower right corner of the notification 7170 occludes the representation 7014′, and so the lower right corner of the notification 7170 is displayed with a different appearance (e.g., as shown by the shading the lower right corner of the notification 7170 in
In some embodiments, the different appearance is a different brightness (e.g., the lower right corner appears lighter or darker than the portions of the notification 7170 and/or the notification 7172 that do not occlude the representation 7014′). In some embodiments, the different appearance is a different color (e.g., the lower right corner has a different color than the portions of the notification 7170 and/or the notification 7172 that do not occlude the representation 7014′). In some embodiments, the different appearances are based at least in part on a visual characteristic of the representation 7014′ (e.g., the different appearance is a different color, and the different color is selected based on the color(s) of the portion of the representation 7014′ that is occluded by the notification 7170 and/or the notification 7172).
A bottom portion of the group 7174 occludes the representation 7014′, and so the bottom portion of the group 7174 is also displayed with a different appearance than other portions of the group 7174. In some embodiments, the lower right corner of the notification 7170 and the bottom portion of the group 7174 have the same appearance (e.g., the same darkness and/or the same color).
The group 7174 occludes the virtual object 7012, and so the lower right corner of the group 7174 is displayed with a different appearance (e.g., darker than and/or a different color than) than other portions of the group 7174. In some embodiments, the lower right corner of the group 7174 is displayed with a different appearance (e.g., darker than) than the other portions of the group 7174, and the different appearance is also different from the appearance of the lower right corner of the group 7174 (e.g., the lower right corner of the group 7174 is displayed with an appearance that is not as dark as the bottom portion of the notification 7172 and/or the lower right corner of the notification 7170 (e.g., because the representation 7014′ has a darker appearance than the virtual object 7012)). In some embodiments, the difference in darkness between the lower right corner of the group 7174 and the bottom portion of the notification 7172 and/or the lower right corner of the notification 7170 is proportional to the difference in darkness between the virtual object 7012 and the representation 7014′.
In some embodiments, the different appearance is a different brightness (e.g., the lower right corner appears lighter or darker than the portions of the group 7174 that do not occlude the virtual object 7012). In some embodiments, the different appearance is a different color (e.g., the lower right corner has a different color than the portions of the group 7174 that do not occlude the virtual object 7012). In some embodiments, the different appearances are based at least in part on a visual characteristic of the virtual object 7012 (e.g., the different appearance is a different color, and the different color is selected based on the color(s) of the portion of the virtual object 7012 that is occluded by the group 7174).
In
The computer system 7100 also displays the affordance 7182, as well as a “Back” affordance 7188, in accordance with some embodiments. In response to detecting that the user’s attention 7116 is directed to the “Back” affordance 7188 (e.g., in combination with an air gesture such as an air tap or an air pinch or another selection gesture), and as shown in
In some embodiments, the computer system 7100 displays an expanded version of the notification 7170 (or the notification 7178), in response to detecting that the user’s attention 7116 is directed to the notification 7170 (or the notification 7178), in a similar fashion as described above with reference to the notification 7172.
In response to detecting that the user’s attention 7116 is directed to the group 7174, and as shown in
In some embodiments, only notifications corresponding to the third application (e.g., Application 3) are displayed (e.g., in
In some embodiments, the notification 7174-a and the notification 7174-b are displayed (in
In some embodiments, the computer system 7100 displays additional notifications corresponding to the third application (Application 3). In some embodiments, the computer system 7100 displays up to a maximum number of notifications (e.g., a maximum of four notifications) for the third application. In some embodiments, the maximum number of notifications for the third application is the same as the maximum number of notifications for the notification history user interface (e.g., the maximum as described above with reference to
In response to detecting that the user’s attention 7116 is directed to a “Back” affordance 7190, and as shown in
In response to detecting that the user’s attention 7116 is directed to a location that is near the right edge of the display of the computer system 7100, and as shown in
In
The group 7174 scrolls to the left, to the position previously occupied by the notification 7172 (e.g., in
A notification 7192 for a fifth application (e.g., Application 5, which is different from Applications 1-4) is displayed at the location previously occupied by the notification 7178 (e.g., in
In some embodiments, an indicator (e.g., similar to the indicator 7180, but flipped along the horizontal axis) is displayed behind the notification 7172 to indicate that the notification 7170 has been scrolled off the display. In some embodiments, the notification 7170 can be redisplayed (e.g., the notifications can be scrolled to the right), for example, in response to detecting that the user’s attention is directed to a location that is near the right edge of the display of the computer system 7100 (or any of the alternatives discussed above with reference to the user’s attention in
In
In some embodiments, the user 7002 performs an analogous input (e.g., directing the user’s attention 7116 to a notification or group, and performing an air gesture such as an air tap or an air pinch, as described above with reference to
In some embodiments, even if the group representation represents multiple notifications, the entire group representation (e.g., the group 7174, in its entirety) is dismissed in response to detecting user input directed to the group representation. In some embodiments, the computer system 7100 ceases to display a first notification represented by the group 7174 if the user input meets first criteria (e.g., is a first type of input, such as an air gesture (e.g., an air tap or an air pinch)), and the computer system 7100 ceases to display the group representation (e.g., all notifications represented by the group 7174) if the user input meets second criteria different from the first criteria (e.g., is a second type of input, different from the first input). In some embodiments, the second criteria are an extension of the first criteria (e.g., the first criteria are met if the user’s attention is directed to the group representation for a first threshold amount of time, and the second criteria are met if the user’s attention remains directed to the group representation for a second threshold amount of time that is longer than the first threshold amount of time (where both the first threshold amount of time and the second threshold amount of time are measured from the same starting point (e.g., a time when the computer system 7100 first detects that the user’s attention is directed to the group representation)).
In some embodiments, a group representation for a group of notifications includes, optionally in a peripheral region of and/or around a representative notification of the group, an affordance for dismissing an individual notification (e.g., the representative notification for the group, or the top notification of the group that is currently displayed) and/or an affordance for dismissing the entire group of notifications. In response to detecting that the user’s attention is directed to the affordance for dismissing individual notification (e.g., in combination with an air gesture (e.g., an air tap or an air pinch), an input via a hardware controller, and/or a verbal input), the computer system 7100 ceases to display an individual notification (e.g., the current notification representing the group of notifications) (e.g., and optionally displays another notification of the group of notifications, as the representative notification for the group of notifications). In response to detecting that the user’s attention is directed to the affordance for dismissing the entire group of notifications (e.g., in combination with an air gesture (e.g., an air tap or an air pinch), an input via a hardware controller, and/or a verbal input), the computer system 7100 ceases to display the group representation (e.g., and all notifications represented by the group representation), in accordance with some embodiments.
As shown in
In some embodiments, the notifications and groups in the notification history user interface are environment-locked, and do not move in accordance with movement of the user (e.g., or changes to the viewpoint of the user). For example, the notifications and groups in
In some embodiments, the notifications and groups in the notification user interface cannot be manually positioned by the user (e.g., the computer system 7100 determines the locations of the notifications and groups in the notification user interface, and those locations cannot be modified by the user 7002 (e.g., via gaze inputs, air gestures, and/or movement)). In some embodiments, the notifications and groups in the notification user interface cannot be manually positioned by the user, regardless of whether the notifications and groups are viewpoint-locked or environment-locked.
Additional descriptions regarding
In some embodiments, a respective type of user interface objects has a corresponding hiding behavior, such that one or more of the user interface objects automatically, without user input, cease to be displayed in accordance with respective hiding criteria being met. For example, notification 7196 continues to be displayed until a threshold amount of time has passed (e.g., 30 seconds, 1 minute, 5 minutes, or another amount of time), and, after the threshold amount of time has passed, notification 7196 automatically ceases to be displayed, and/or notification 7196 continues to be displayed until the user interacts with (e.g., directs the user’s attention to) the notification 7196. In some embodiments, alert 7198-1 continues to be displayed until a user input is detected for dismissing the alert 7198-1. In some embodiments, system function menu 7024 continues to be displayed while the user’s attention is directed to (e.g., or in an area around) system function menu 7024, and automatically ceases to be displayed in accordance with a determination that the user’s attention is not directed to system function menu 7024, as described above. In some embodiments, the user interface object for virtual assistant 7200-1 is displayed for a threshold amount of time (e.g., 20 seconds, 1 minute, or another amount of time) before automatically ceasing to be displayed, is displayed until the user has interacted with (e.g., and/or dismisses) the virtual assistant, and/or is displayed until the user moves (e.g., and/or changes the user’s viewpoint by a threshold amount). In some embodiments, after ceasing to be displayed in the three-dimensional environment, user interface objects of the different types are optionally redisplayed at a same or a different position in the three-dimensional environment, in accordance with respective redisplay criteria being met.
In some embodiments, a respective type of user interface objects has a corresponding follow behavior such that one or more of the user interface objects automatically, without user input, moves and follows the movement of the user and/or movement of the viewpoint of the user, in accordance with respective criteria being met. In some embodiments, different types of objects have different behaviors with respect to whether and/or how to move in accordance with movement of the user and/or movement of the viewpoint of the user. For example, in some embodiments, objects of a first type exhibit a continuous follow behavior, objects of a second type exhibit a delayed follow behavior, objects of a third type do not follow but instead are dismissed when certain movement of the user or viewpoint thereof is detected and are redisplayed in response to an explicit reinvocation input, and other objects (e.g., objects not of the first, second or third type) are world-locked and do not follow or get dismissed from the three-dimensional environment in response to movement of the user or viewpoint thereof. In some embodiments, during movement of the user and/or viewpoint thereof, a user interface object exhibiting a continuous follow behavior is always within a currently displayed view of the three-dimensional environment, while a user interface object exhibiting a delayed follow behavior may cease to be visible in the currently displayed field of view (e.g., the user interface object has not moved or has not caught up with the movement of the viewpoint), and the user interface object is displayed in the currently displayed view of the three-dimensional environment after the viewpoint has stopped moving (e.g., moved less than a threshold amount in a unit of time) for at least a threshold amount of time, in accordance with respective redisplay criteria being met. In some embodiments, not all of the user interface objects described herein are displayed concurrently. In some embodiments, user interface objects of a subset of the different types of user interface objects are concurrently displayed in the same view of the three-dimensional environment, and exhibit different behaviors in accordance with the movement of the user or viewpoint thereof. For example, although
It will be understood that although the user interface objects in
It will also be understood that while the examples below describe certain types of user interface objects having particular movements and/or follow behaviors, in some embodiments, the user interface objects (e.g., and/or other user interface objects) may have a different type of movement and/or follow behavior in a different embodiment. For example, in the examples described below, notification 7196 is described as being of the first type of user interface object, and thus having a continuous follow behavior, but could also (e.g., or instead) be considered a different type of user interface object, and thus have a different follow behavior described herein in accordance with some embodiments. For example, notification 7196 is optionally assigned as a second type of user interface object that corresponds to a delayed follow behavior, wherein the notification 7196 is dismissed or hidden and, is optionally respawned in another view (e.g., the settled view).
In some embodiments, the first type of the plurality of types of user interface objects corresponds to user interface objects that have the continuous follow behavior. For example, user interface objects of the first type continue to be displayed in the current view of the user as the user moves or otherwise changes the user’s current viewpoint of the three dimensional environment. In some embodiments, user interface objects of the first type include one or more of a system function menu (e.g., system function menu 7024, or other system function menus), notifications for a plurality of applications (e.g., including notification 7196, and/or other notifications), and closed captions (e.g., textual description or transcription displayed for media content playing back in an application window).
In some embodiments, different categories of user interface objects of the first type have different manners of continuous follow behavior. For example, notification 7196 and system function menu 7024 are both user interface objects of the first type that have continuous follow behavior (e.g., and optionally continuously follow the user’s viewpoint in different manners). For example, how closely a respective user interface object of the first type follows the movement of the viewpoint of the user is based on the category of the user interface object. For example, the amount of movement and/or rate of movement of notification 7196 and system function menu 7024 (e.g., and/or for other user interface objects in the first category, such as closed captions) are different from each other, for the same movement of the viewpoint of the user in the three-dimensional environment, as described below. In some embodiments, the criteria for triggering the initiation of the continuous follow behavior are optionally different for different categories of user interface objects that exhibit the continuous follow behavior. For example, the computer system optionally implements different distance, angle, and/or speed requirements different categories of the user interface objects exhibiting the continuous follow behavior to start following the viewpoint in response to the movement of the viewpoint. In some embodiments, different categories of user interface objects that exhibit continuous follow behavior also have different simulated movement characteristics (e.g., simulated inertia, movement paths, delay distance, delay time, catchup rate, and other characteristics).
In some embodiments, a respective user interface object (e.g., and/or a respective category of user interface object) of the first type is assigned a damping factor that defines a rate at which the respective user interface object follows (e.g., and/or lags behind) the user’s movements that change the user’s viewpoint. For example, in
In
In some embodiments,
In some embodiments, user interface objects of the second type include one or more of a setup user interface (e.g., that is displayed to setup the computer system and optionally includes one or more controls for settings/preferences and/or informational tips to navigate the computer system during an initial setup process), a system alert (e.g., alert 7198-1, or other alerts related to system status changes or alerts that require the user’s attention or input in order to be dismissed), and a media player window that plays back media content (e.g., a mini media player window that has a reduced set of controls as compared to a regular media player window) that displays media content. In some embodiments, the media player window (e.g., mini media player window) is concurrently displayed with an application window (e.g., that is world-locked), at a size that is smaller than the application window (e.g., the media player is a miniature window that continues to playback media content as the user’s viewpoint changes), such that, in response to the user changing viewpoints, and upon the user settling at a current viewpoint, the media player window is respawned at a same position relative to the user’s current viewpoint without updating (e.g., respawning) a position of the application window (e.g., the application window remains at a same position in the three dimensional environment). In some embodiments, the mini media player window may appear as overlaid on top of other content and windows in the three-dimensional environment, even as the user interacts with the other content and/or windows.
In some embodiments, user interface objects of the second type demonstrate delayed follow behavior, but different categories of the user interface objects of the second type optionally exhibit different delayed follow behaviors. For example, a system alert is an example of user interface objects in a first category of the second type of user interface objects, and the system alert ceases to be displayed while the user’s viewpoint changes (e.g., before the user’s viewpoint settles), while alert 7198-2 (e.g., which is in a second category of the second type of user interface objects) is faded and/or blurred while the user’s viewpoint changes, and the media player window (e.g., which is in a third category of the second type of user interface objects) optionally fades and/or blurs out by a different amount than alert 7198-2.
In some embodiments, the different user interface objects of the second type are respawned at different times based on when the user settles at the new viewpoint, as described below with respect to
For example, in
In some embodiments, the user interface object for virtual assistant 7200-1 is an example of the third type of user interface objects (e.g., dismissed user interface objects) that do not exhibit a follow behavior, but when re-invoked by an explicit user input, appear at a new position within the currently displayed view of the three-dimensional environment. For example, the user interface object for virtual assistant 7200-1 optionally ceases to be displayed as the user’s viewpoint moves, and is not redisplayed upon the user’s viewpoint settling in
In some embodiments, as the user’s viewpoint continues to change, for example in
In some embodiments, as the user continues to move in the physical environment, and the viewpoint of the user continues to change (e.g., and has not yet settled), alert 7198-4 is displayed with more visual deemphasis and/or optionally ceases to be displayed. For example, in
In some embodiments, in response to determining that the user’s viewpoint is settled, as illustrated in
In some embodiments, a second user interface object of the second type, such as a media player window (e.g., a mini media player window, or a pinned object or always-on-top object) is displayed in the view of the three-dimensional environment illustrated in
In some embodiments, redisplaying alert 7198-1 upon determining that the viewpoint of the user is settled in
In some embodiments, user interface objects of the third type (e.g., virtual assistant 7200-2, or another user interface object) are not redisplayed in response to detecting that the user’s viewpoint is settled. For example, in
In some embodiments, the different types of user interface objects described above (e.g., first type having continuous follow behavior, second type having delayed follow behavior, and third type not having follow behavior) are associated with different criteria for dismissing the respective user interface object. For example, a respective user interface object is associated with a respective safe zone based on the type of user interface objects to which the respective user interface object belongs (e.g., wherein the dismiss criteria is based at least in part on whether the user’s viewpoint has a location that corresponds to an area within the safe zone and/or outside of the safe zone). In some embodiments, the respective safe zone of a user interface object is based on a head position (e.g., or position of another body part, such as a torso position) of the user and/or based on the current view when the respective user interface object is initially displayed (e.g., spawned into the current view). For example, the initial position at which virtual assistant 7200-2 is spawned (e.g., in a bottom center of the view) is used to define the safe zone for virtual assistant 7200-2, including, for example, a U-shaped area of the three dimensional environment that includes virtual assistant 7200-2.
In some embodiments, while the user’s viewpoint has a location that corresponds to a portion of the three dimensional environment that is within the safe zone for the user interface object, the user interface object does not move its position within the three dimensional environment. In some embodiments, if the user’s viewpoint has a location that is outside of the safe zone, the user interface object is subject to the continuous follow, delayed follow, and/or dismiss behavior described above with reference to
In some embodiments, the safe zone for the third type of user interface objects is an area that comprises a U-shape, or another irregular shape (e.g., as opposed to a spherical shape, a box shape, or other regular shapes). For example, while the user’s current viewpoint has a location that corresponds to an area that is within the safe zone of the third user interface object (e.g., within a U-shaped area defined relative to virtual assistant 7200-2), virtual assistant 7200-2 is not dismissed in response to detecting the user’s head has turned to a position within the safe zone. For example, if the U-shaped area spans to the left and right of virtual assistant 7200-2, if the user’s head turns to the right and/or left such that the user’s viewpoint is directed to the U-shaped area, virtual assistant 7200-2 remains at its position (e.g., as if world-locked), whereby the user is enabled to approach and/or interact with (e.g., using touch inputs or other inputs) virtual assistant 7200-2. In some embodiments, in response to detecting that the user’s current viewpoint has a location that corresponds to an area that is outside the safe zone of the third user interface object (e.g., outside of a U-shaped area defined relative to virtual assistant 7200-2), virtual assistant 7200-2 is dismissed, as described with reference to
In some embodiments, while the viewpoint of the user is in an area within the respective safe zone for the respective user interface object (e.g., a user interface object of the first and/or second type), the respective user interface object does not follow the movement of the user’s viewpoint (e.g., the user interface object is not displayed with continuous and/or delayed follow behavior). As such, while the user’s viewpoint has a location that corresponds to an area within the safe zone, alert 7198-1 and notification 7196 do not follow the user’s viewpoint (e.g., and appear world-locked). In some embodiments, if the user’s viewpoint has a location that corresponds to an area outside of the respective safe zone for the respective type of user interface object, the respective user interface object (e.g., alert 7198-1 and/or notification 7196) is displayed with the follow behaviors described above with reference to
Additional descriptions regarding
Conditionally displaying (e.g., based on the attention criteria) the plurality of affordances for accessing system functions of a computer system in response to detecting the user’s attention directed to the first user interface object reduces the number of inputs needed to access system functions of the computer system without cluttering the user interface with additional displayed controls, in accordance with some embodiments.
The computer system displays (8002), via the first display generation component, a first user interface object (e.g., indicator 7010 of system function menu (
While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects (8004), via the one or more input devices, movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the physical environment. In response to detecting the movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the physical environment, the computer system maintains (8006) display of the first user interface object at a respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user (or, alternatively, relative to the viewport through which the three-dimensional environment is visible) while a second view of the three-dimensional environment is visible (e.g., the second view is different from the first view). In some embodiments, the first user interface object is maintained at a respective position in the three-dimensional environment having the first spatial arrangement relative to the viewpoint of the user (or alternatively, relative to the viewport through which the three-dimensional environment is visible). In some embodiments, the first user interface object is maintained at the first location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment. For example, as the viewpoint of user 7002 changes from
While the second view of the three-dimensional environment is visible and while displaying the first user interface object in the second view of the three-dimensional environment, the computer system detects (8008), via the one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the first gaze input directed to the first user interface object, and in accordance with a determination that the first gaze input satisfies attention criteria with respect to the first user interface object, the computer system displays (8010) a plurality of affordances for accessing system functions (e.g., performing system operations) of the first computer system (e.g., a volume control, a search affordance, a notification center, a control panel, and/or a virtual assistant). For example, as shown in and described herein with reference to
In some embodiments, the computer system detects that the first gaze input is directed to a first region in the second view of the three-dimensional environment, wherein the first region corresponds to the position of the first user interface object in the second view of the three-dimensional environment (e.g., the first region is defined to be within or substantially within a visible region surrounding the first user interface object in the second view of the three-dimensional environment), and the computer system detects that the first gaze input is directed to the first region for at least a first amount of time (e.g., 500 milliseconds, 700 milliseconds, 1 second, or two seconds). In some embodiments, the first region encompasses one or more locations in the second view of the three-dimensional environment that are within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of the first user interface object. In some embodiments, detecting the first gaze input directed to the first user interface object includes determining that the first gaze input remains directed to one or more locations within the first region for at least the first amount of time (e.g., without being directed to any location outside of the first region during the first amount of time). For example, as described above with reference to
In some embodiments, the first user interface object is translucent and has an appearance that is based on at least a portion of the three-dimensional environment over which the first user interface object is displayed. In some embodiments, the portion of the three-dimensional environment over which the first user interface object is displayed includes a representation of a physical environment in a field of view of one or more cameras of the first computer system (e.g., real world content). In some embodiments, the portion of the three-dimensional environment over which the first user interface object is displayed includes computer-generated content (e.g., virtual content not corresponding to the physical environment, such as virtual content other than a representation of the physical environment captured by one or more cameras of the first computer system). For example, as described above with reference to
In some embodiments, the computer system: displays the first user interface object with a first appearance at the first position in the three-dimensional environment, wherein the first appearance of the first user interface object at the first position is based at least in part on a characteristic of the three-dimensional environment at the first position in the first view of the three-dimensional environment (e.g., the first user interface is translucent and has a first appearance based on the physical environment or virtual content that is behind the first user interface object in the first view of the three-dimensional environment). In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system displays the first user interface object with a respective appearance (e.g., different from the first appearance) at the respective position in the three-dimensional environment that has the first spatial arrangement relative to the respective portion of the user, wherein the respective appearance of the first user interface object at the respective position is based at least in part on a characteristic of the three-dimensional environment at the respective position (e.g., after movement of viewpoint of the user, the first user interface object has a respective appearance based on the physical environment or virtual content that is behind the first user interface object in the second view of the three-dimensional environment, which in some cases is different from the first appearance due to the movement of the viewpoint and differences in the three-dimensional environment between the previous and current locations of the first user interface object). For example, as described above with reference to
In some embodiments, the computer system displays the first user interface object with a first level of prominence at the first position in the three-dimensional environment. In some embodiments, maintaining display of the first user interface object at the respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user (or alternatively, having the first spatial arrangement relative to the viewport through which the three-dimensional environment is visible) while the second view of the three-dimensional environment is visible includes: displaying the first user interface object with a second level of prominence, different from the first level of prominence (e.g., the second level of prominence is less prominent or visually deemphasized relative to the first level of prominence), during the movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment; and displaying the first user interface object with the first level of prominence at the respective position in the three-dimensional environment that has the first spatial arrangement relative to the respective portion of the user. In some embodiments, the first user interface object ceases to be displayed (e.g., fades out entirely, instead of being displayed with the second level of prominence) during the movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, and the first user interface object is displayed (e.g., after the movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, while the viewpoint of the user remains substantially at or within a threshold range of the second viewpoint (e.g., within the lazy follow threshold)) with the first level of prominence at the respective position in the three-dimensional environment that has the first spatial arrangement relative to the respective portion of the user. This is described above, for example, in the description of
In some embodiments, while the first user interface object has a first appearance (e.g., an appearance that indicates that there is no notification that satisfies the timing criteria), the computer system detects that a first event for a first notification satisfies timing criteria (e.g., a notification has been generated within a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds)). In some embodiments, in response to detecting that the first event for the first notification satisfies timing criteria, the computer system changes an appearance of the first user interface object from the first appearance to a second appearance, wherein the second appearance includes an indication of a respective application associated with the first notification. This is shown, for example, in
In some embodiments, in accordance with a determination that the first notification is associated with a first application, the second appearance includes an indication of the first application, and in accordance with a determination that the first notification is associated with a second application different from the first application, the second appearance includes an indication of the second application. This is described above, for example, in the description of
In some embodiments, the computer system displays, via the first display generation component, a second user interface object (e.g., an affordance, an application icon, or an interactive object) while the first view of the three-dimensional environment is visible. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via the one or more input devices, a first user input (e.g., an air gesture (e.g., an air tap or an air pinch), a gaze input, a verbal input, and/or a combination of a gaze input, hand gesture, and/or verbal input) interacting with the second user interface object. In some embodiments, the second user interface object is a virtual object, and the first user input interacts with the second user interface object by changing a size, position, and/or orientation of the second user interface object. In some embodiments, the second user interface object is an affordance, and the first user input interacts with the second user interface object by activating the second user interface object. In some embodiments, the second user interface object is a user interface object (e.g., a slider or a dial) for adjusting a value (e.g., for a setting) of the first computer system, and the first user input interacts with the second user interface object by moving (e.g., moving a slider, or rotating a dial) the second user interface object. In some embodiments, while detecting the first user input interacting with the second user interface object, the computer system maintains display of the first user interface object at the respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user while the first view of the three-dimensional environment is visible (e.g., the first user interface object continues to be displayed even when the user interacts with the second user interface object (e.g., another user interface object or control)). This is described above, for example, in the description of
In some embodiments, the computer system displays the first user interface object at a second position in the second view of the three-dimensional environment. The computer system detects, via the one or more input devices, an input that corresponds to movement of the viewpoint of the user from a second viewpoint to a third viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the second viewpoint to the third viewpoint in the three-dimensional environment: in accordance with a determination that the movement of the viewpoint of the user from the second viewpoint to the third viewpoint does not satisfy a threshold amount (e.g., threshold angle (e.g., 5 degrees, 10 degrees, 25 degrees, 45 degrees, 90 degrees, 120 degrees, or any threshold angle between 0 and 120 degrees) and/or distance (e.g., 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters)) of movement, the computer system maintains display of the first user interface object at the second position in the three-dimensional environment (e.g., such that the first user interface object no longer has the first spatial arrangement relative to the respective portion of the user); and in accordance with a determination that the movement of the viewpoint of the user from the second viewpoint to the third viewpoint satisfies the threshold amount of movement, the computer system ceases to display the first user interface object at the second position in the three-dimensional environment, and the computer system displays the first user interface object at a third position in the three-dimensional environment, wherein the third position in the three-dimensional environment has the first spatial arrangement relative to the respective portion of the user (e.g., relative to the third viewpoint of the user). In some embodiments, the first user interface object is displayed at the same location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment (e.g., the first and second locations are the same location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment). This is shown in
In some embodiments, the computer system displays an animated transition of the plurality of affordances for accessing system functions of the first computer system expanding downward from the first user interface object (e.g., in response to the first gaze input directed to the first user interface object and in accordance with the determination that the first gaze input satisfies the attention criteria with respect to the first user interface object). In some embodiments, after displaying the animated transition of the plurality of affordances for accessing system functions of the first computer system expanding downward from the first user interface object, the computer system ceases to display the first user interface object. In some embodiments, the computer system concurrently displays the animated transition of the plurality of affordances for accessing system functions of the first computer system expanding downward from the first user interface object, and an animated transition of the first user interface object (e.g., fading out of view, and/or collapsing upward/downward/inward). In some embodiments, the plurality of affordances for accessing system functions of the first computer system are displayed (e.g., remain displayed) below (e.g., immediately below, within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of) the location of first user interface object before the first user interface object ceased to be displayed. This is described above, for example, in the description of
In some embodiments, the computer system displays an animated transition of the plurality of affordances for accessing system functions of the first computer system gradually appearing (e.g., fading in, and/or expanding) (e.g., in response to the first gaze input directed to the first user interface object and in accordance with the determination that the first gaze input satisfies the attention criteria with respect to the first user interface object). This is described above, for example, in the description of
In some embodiments, while displaying the plurality of affordances for accessing system functions of the first computer system, the computer system detects that the first gaze input no longer satisfies the attention criteria with respect to the first user interface object, and in some embodiments, in response to detecting that the first gaze input no longer satisfies the attention criteria with respect to the first user interface object, the computer system ceases to display the plurality of affordances for accessing system functions of the first computer system. In some embodiments, the attention criteria require that the first gaze input be directed to the first user interface object (e.g., the attention criteria require that the user be gazing or looking at the first user interface object), and the first gaze input ceases to satisfy the attention criteria if the user looks away from the first user interface object. This is described above, for example, in the description of
In some embodiments, the first position is in a periphery region (e.g., is within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of an edge) of the first display generation component (e.g., relative to a field of view of a user using the first display generation component, such as a user’s field of view while wearing a head-mounted display). This is shown in
In some embodiments, the computer system detects, via the one or more input devices, a verbal input by the user. In some embodiments, in response to detecting the verbal input by the user, in accordance with a determination that the verbal input is detected while the user’s gaze is directed to a respective affordance of the plurality of affordances for accessing system functions of the first computer system (e.g., the user gazes at the respective affordance and speaks a voice command) (e.g., and in accordance with a determination that the respective affordance can be voice activated), the computer system performs a function associated with the respective affordance. In some embodiments, in accordance with a determination that the respective affordance is not voice activatable, the function associated with the respective affordance is not performed in response to detecting the verbal input (e.g., even if the verbal input is detected while the user’s gaze is directed to the respective affordance). In accordance with a determination that the verbal input is detected while the user’s gaze is directed to a first affordance of the plurality of affordances for accessing system functions of the first computer system (e.g., the user gazes at the first affordance and speaks a voice command) (e.g., and in accordance with a determination that the first affordance can be voice activated), performing a first function associated with the first affordance. In accordance with a determination that the verbal input is detected while the user’s gaze is directed to a second affordance, different from the first affordance, of the plurality of affordances for accessing system functions of the first computer system (e.g., the user gazes at the second affordance and speaks a voice command) (e.g., and in accordance with a determination that the second affordance can be voice activated), performing a second function associated with the second affordance, where the second function is different from the first function (e.g., performing a system search operation instead of providing a voice input to a virtual assistant). This is shown in
In some embodiments, while detecting the verbal input by the user, the computer system displays a first visual indication (e.g., highlighting of the respective affordance, and/or displaying an animation associated with the respective affordance) corresponding to the verbal input by the user (e.g., while the user is speaking) (e.g., indicative of verbal input being detected). In some embodiments, displaying the first visual indication includes, in accordance with a determination that attention of the user is directed to a first affordance, displaying the first visual indication with respect to (e.g., by changing an appearance of the first affordance and/or displaying a visual indication over or near the first affordance) the first affordance (e.g., and not with respect to a second affordance), and in accordance with a determination that attention of the user is directed to a second affordance, displaying the first visual indication with respect to (e.g., by changing an appearance of the second affordance and/or displaying a visual indication over or near the second affordance) the second affordance (e.g., and not with respect to the first affordance). This is shown in
In some embodiments, the computer system displays text corresponding to the verbal input by the user (e.g., the respective affordance is for accessing a search function of the first computer system, and the computer system displays text corresponding to a spoken search term or query in a search bar or text entry field displayed adjacent to (e.g., immediately below) the first user interface object; optionally the search bar or text entry field is displayed in response to one or more inputs directed to the respective affordance). This is shown in
In some embodiments, the computer system displays the first visual indication with a characteristic value, wherein the characteristic value (e.g., highlighting, size, and/or opacity) is based at least in part on an audio level (e.g., a volume, frequency, and/or tone) of the verbal input by the user (e.g., the first visual indication changes appearance as the volume of the verbal input by the user changes). This is described above, for example, in the descriptions of
In some embodiments, the computer system detects, via the one or more input devices, a first hand gesture (e.g., an air gesture, such as an air tap or an air pinch, or another selection input) by the user. In some embodiments, in response to detecting the first hand gesture by the user, and in accordance with a determination that the first hand gesture by the user is detected while the user’s gaze is directed to a respective affordance of the plurality of affordances for accessing system functions of the first computer system (e.g., the user performs the hand gesture while gazing at the respective affordance), the computer system performs a function associated with the respective affordance (e.g., displaying a system user interface for a system function that is accessed via the respective affordance). In some embodiments, the system functions of the first computer system include one or more of: a volume control function, a search function, a function for displaying notifications received by the first computer system (e.g., a notification center function), a function for displaying a user interface (e.g., a control panel user interface) that includes a plurality of selectable user interface elements for controlling additional system functions of the first computer system, and/or a virtual assistant function. This is shown in
In some embodiments, the computer system displays a respective affordance of the plurality of affordances with a first appearance. The computer system detects, via the one or more input devices, a change in configuration of the user’s hand (e.g., into a ready state configuration). In some embodiments, in response to detecting the change in configuration of the user’s hand, and in accordance with a determination that the user’s hand meets first criteria, the computer system displays the respective affordance with a second appearance different from the first appearance. In some embodiments, the first appearance provides visual feedback that the user’s hand does not meet the first criteria, and the second appearance provides visual feedback that the user’s hand meets the first criteria. For example, the respective affordance with the second appearance has a different size, moves closer to or further from the viewpoint of the user (e.g., along a z-axis), is changed in thickness (e.g., z-axis height), has a thicker (or thinner) border, and/or has a different color, compared to the respective affordance with the first appearance. In some embodiments, the respective affordance with the second appearance has a shadow, has a glow (e.g., has increased brightness and/or appears to emit light), and/or is animated, while the respective affordance with the first appearance does not.
In some embodiments, displaying the plurality of affordances for accessing system functions of the first computer system includes: in accordance with a determination that the user’s hand is in a ready state configuration, displaying a first affordance of the plurality of affordances with first appearance; and in accordance with a determination that the user’s hand is not in the ready state configuration, displaying the first affordance with a second appearance that is different from the first appearance. In some embodiments, for a second affordance of the plurality of affordances, the appearance of the second affordance changes in response to detecting the change in the configuration of the user’s hand and in accordance with the determination that the user’s hand meets the first criteria. In some embodiments, displaying the respective affordance with the second appearance includes: in accordance with a determination that the user’s attention is directed to a first affordance when the user’s hand meets the first criteria, changing the first affordance from the first appearance to the second appearance (e.g., without changing one or more other affordances from the first appearance to the second appearance) and in accordance with a determination that the user’s attention is directed to a second affordance when the user’s hand meets the first criteria, changing the second affordance from the first appearance to the second appearance (e.g., without changing one or more other affordances, such as the first affordance, from the first appearance to the second appearance). In some embodiments, if the attention of the user shifts from a first affordance to a second affordance while the user’s hand meets the first criteria, the first affordance shifts from the second appearance to the first appearance and the second affordance shifts from the first appearance to the second appearance. For example, as described herein with reference to
In some embodiments, the computer system displays a second affordance of the plurality of affordances for accessing system functions of the first computer system with a third appearance. The computer system detects, via the one or more input devices, a change in position of the user’s hand. In some embodiments, in response to detecting the change in position of the user’s hand, and in accordance with a determination that the user’s hand meets the first criteria, the computer system maintains display of the second affordance with the third appearance. In some embodiments, in accordance with a determination that the user’s hand does not meet the first criteria, the computer system displays the second affordance with the third appearance (e.g., the second affordance is displayed with the third appearance without regard to whether the user’s hand meets the first criteria) (e.g., concurrently with displaying the first affordance with the second appearance different from the first appearance in response to detecting the change in position of the user’s hand such that the user’s hand meets the first criteria). This is described above, for example, with reference to
In some embodiments, after (e.g., while) displaying the plurality of affordances for accessing system functions of the first computer system, the computer system detects, via the one or more input devices, a second gaze input directed to a respective affordance of the plurality of affordances for accessing system functions of the first computer system. In some embodiments, in response to detecting the second gaze input directed to the respective affordance, the computer system displays additional content associated with the respective affordance (e.g., a description of setting(s) associated with the respective affordance, instructions for adjusting the setting(s) associated with the respective affordance, and/or one or values associated with the setting(s) associated with the respective affordance). In some embodiments, the additional content associated with the respective affordance is displayed after a delay (e.g., 1 second, 5 seconds, 10 seconds, or any threshold amount of time between 0 and 10 seconds). In some embodiments, in response to detecting that the second gaze input is no longer directed to the respective affordance, the computer system ceases to display the additional content associated with the respective affordance). This is described above, for example, with reference to
In some embodiments, the computer system detects, via the one or more input devices, a second user input to activate a first affordance of the plurality of affordances for accessing system functions of the first computer system. In some embodiments, in response to detecting the second user input to activate the first affordance of the plurality of affordances for accessing system functions of the first computer system, the computer system displays a first system user interface for a respective system function of the first computer system (e.g., associated with the first affordance). This is shown in
In some embodiments, while displaying the first system user interface, the computer system detects, via the one or more input devices, a third user input to activate a second affordance, different from the first affordance, of the plurality of affordances for accessing system functions of the first computer system. In some embodiments, in response to detecting the third user input to activate the second affordance of the plurality of affordances for accessing system functions of the first computer system, the computer system displays a second system user interface, different from the first system user interface, for a respective system function of the first computer system (e.g., associated with the second affordance), and the computer system ceases to display the first system user interface (e.g., associated with the first affordance). In some embodiments, the computer system replaces display of the first system user interface with display of the second system user interface. This is described above, for example, in the description of
In some embodiments, while displaying the first system user interface (e.g., concurrently with displaying the plurality of affordances for accessing system functions), and in accordance with a determination that the user’s gaze is directed to the plurality of affordances for accessing system functions of the first computer system, the computer system visually deemphasizes (e.g., by blurring, fading, and/or shrinking) the first system user interface (e.g., relative to the plurality of affordances for accessing system functions). This is described above, for example, in the description of
In some embodiments, while displaying the plurality of affordances for accessing system functions of the first computer system (e.g., concurrently with displaying the first system user interface), and in accordance with a determination that the user’s gaze is directed to the first system user interface, the computer system visually deemphasizes (e.g., blurring, fading, and/or shrinking) the plurality of affordances for accessing system functions of the first computer system (e.g., relative to the first system user interface). This is described above, for example, in the description of
In some embodiments, the computer system displays, via the first display generation component, an application launching user interface (e.g., a home user interface) in the three-dimensional environment (e.g., the second user input is detected while the application launching user interface is displayed). In some such embodiments, displaying the first system user interface includes replacing display of the application launching user interface with display of the first system user interface (e.g., ceasing to display the application launching user interface). This is described above, for example, with reference to
In some embodiments, before detecting the second user input to activate the first affordance of the plurality of affordances for accessing system functions of the first computer system, the computer system displays, via the first display generation component, an application user interface in the three-dimensional environment (e.g., the second user input is detected while the application user interface is displayed). In some such embodiments, displaying the first system user interface includes displaying the first system user interface over (and/or in front of) the application user interface. This is described above, for example, in the description of
In some embodiments, the computer system detects, via the one or more input devices, a fourth user input that activates a respective affordance of the plurality of affordances for accessing system functions of the first computer system. In some embodiments, in response to detecting the fourth user input that activates the respective affordance of the plurality of affordances for accessing system functions of the first computer system, the computer system displays a respective system user interface for adjusting a respective setting of the first computer system (e.g., changing a volume setting, changing a brightness setting, and/or changing a level of immersion). This is shown in
In some embodiments, the computer system displays the respective system user interface for adjusting the respective setting of the first computer system over at least a portion of one or more of the plurality of affordances. For example, as described above with respect to
In some embodiments, the fourth user input to activate the respective affordance of the plurality of affordances for accessing system functions of the first computer system is a first air pinch gesture. While displaying the respective system user interface for adjusting a respective setting of the first computer system, the computer system detects, via the one or more input devices, a second air pinch gesture (e.g., after releasing the first air pinch gesture), and (e.g., followed by) a change in position of the user’s hand from a first position to a second position (e.g., lateral movement in three-dimensional space, such as an air drag gesture, while the two or more fingers of the user’s hand are in contact with one another). In some embodiments, in response to detecting the second air pinch gesture and the change in position of the user’s hand from the first position to the second position, the computer system adjusts the respective setting of the first computer system in accordance with the change in position of the user’s hand. In some embodiments, adjusting the respective setting of the first computer system includes adjusting the respective setting by an amount based on the amount of movement of the user’s hand. In some embodiments, movement of the user’s hand in a first direction (e.g., toward the right, and/or upward) increases the respective setting, and movement of the user’s hand in another direction (e.g., opposite the first direction, such as toward the left, and/or downward) decreases the respective setting. In some embodiments, the computer system displays a visual indication (e.g., in the respective system user interface, and/or in a user interface displayed concurrently with the respective system user interface and/or the plurality of affordances for accessing system functions of the first computer system) that the respective setting of the first computer system is being adjusted. For example, as described herein with reference to
In some embodiments, the fourth user input that activates the respective affordance of the plurality of affordances for accessing system functions of the first computer system includes a third air pinch gesture, and (e.g., followed by) a change in position of the user’s hand from a first position to a second position (e.g., lateral movement in three-dimensional space). In some embodiments, the respective system user interface for adjusting a respective setting of the first computer system is displayed in response to detecting the third air pinch gesture. In some embodiments, in response to detecting the change in position of the user’s hand from the first position to the second position, the first computer system adjusts the respective setting of the first computer system in accordance with the (e.g., amount of) change in position of the user’s hand. In some embodiments, adjusting the respective setting of the first computer system includes adjusting the respective setting by an amount based on the amount of movement of the user’s hand. In some embodiments, the computer system displays a visual indication (e.g., in the respective system user interface, and/or in a user interface displayed concurrently with the respective system user interface and/or the plurality of affordances for accessing system functions of the first computer system) that the respective setting of the first computer system is being adjusted. This is described herein with reference to
In some embodiments, the computer system displays the plurality of affordances for accessing system functions of the first computer system at a first simulated distance from the viewpoint of the user, wherein the first simulated distance is less than a simulated distance of other user interface objects from the viewpoint of the user (e.g., the first user interface object and/or the plurality of affordances is always displayed “on top” of other user interfaces and/or user interface objects, such that the first user interface object and/or the plurality of affordances is always visible, regardless of what other user interfaces and/or user interface objects are displayed via the first display generation component). In some embodiments, the plurality of affordances is displayed overlaid on at least a portion of the other, further user interface objects in the respective (e.g., currently displayed) view of the three-dimensional environment. This is shown in
In some embodiments, the computer system displays a plurality of system status indicators that include information about a status of the system (e.g., Wi-Fi connection status, cellular connection status, a current time, and/or battery charge state). In some embodiments, the plurality of system status indicators are displayed proximate to (e.g., immediately above, immediately below, immediately to the right or left of) the plurality of affordances for accessing system functions of the first computer system. This is described above, for example, in the description of
In some embodiments, the first user interface object that is displayed while the first view of the three-dimensional environment is visible is displayed while (e.g., and in accordance with a determination that) a third gaze input satisfies first proximity criteria with respect to the first user interface object. While displaying the first user interface object, the computer system detects that the third gaze input ceases to satisfy the first proximity criteria with respect to the first user interface object, and in some embodiments, in response to detecting that the third gaze input ceases to satisfy the first proximity criteria with respect to the first user interface object (e.g., the user’s gaze is directed to a location in the respective view of three-dimensional environment that is greater than a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters) from the first user interface object), the computer system ceases to display the first user interface object. In some embodiments, ceasing to display the first user interface object includes displaying an animated transition (e.g., of the first user interface object fading out of view). This is described above, for example, in the description of
In some embodiments, while the first user interface object is not displayed, (e.g., after ceasing to display the first user interface object) the computer system detects a fourth gaze input that satisfies second proximity criteria with respect to a location at which the first user interface object was previously displayed. In some embodiments, in response to detecting the fourth gaze input that satisfies the second proximity criteria with respect to the location at which the first user interface object was previously displayed (e.g., the user’s gaze is directed to a location in the respective view of three-dimensional environment that is within the threshold distance from the location at which the first user interface object was previously displayed), the computer system displays (e.g., redisplays) the first user interface object. In some embodiments, the second proximity criteria are the same as the first proximity criteria (e.g., the first proximity criteria described above, with reference to ceasing to display the first user interface object in response to detecting that a third gaze input ceases to satisfy the first proximity criteria with respect to the first user interface object). In some embodiments, the second proximity criteria are different from the first proximity criteria (e.g., to add hysteresis, so as to reduce the risk of unintentionally ceasing to display and/or redisplaying the first user interface object when the user’s gaze is near a threshold distance associated with the first and/or second proximity criteria, for example by setting a first threshold distance for satisfying the first proximity criteria to be greater than a second threshold distance for satisfying the second proximity criteria). In some embodiments, redisplaying the first user interface object includes displaying an animated transition (e.g., of the first user interface object fading into view). For example, the first user interface object is displayed in the top center of the field of view of the respective view of the three-dimensional environment (e.g., the top center of a display of the display generation component or of the user’s field of view). When the user looks towards the center of the field of view, the first user interface object ceases to be displayed, and optionally fades out of view. When the user looks upwards towards the top center of the field of view, the first user interface object is redisplayed, and optionally fades into view. This is described above, for example, in the description of
In some embodiments, prior to displaying the first user interface object at the first position in the three-dimensional environment, the computer system detects, via the one or more input devices, a fifth gaze input directed to a second region in the first view of the three-dimensional environment, wherein the second region has a first spatial relationship to the first view of the three-dimensional environment (e.g., the second region is a respective portion of (e.g., an upper left corner, a top center portion, a lower right corner, a peripheral region, and/or another portion) of the field of view provided via the first display generation component that has been consistently associated with displaying the first user interface object, and/or the second region is represented in a respective portion of the respective view (e.g., the first view, or another currently displayed view) of the three-dimensional environment that corresponds to the current viewpoint of the user) as a reactive region for a gaze input that corresponds to a request to display the first user interface object. In response to detecting the fifth gaze input directed to the second region in the first view of the three-dimensional environment, the computer system displays the first user interface object at the first position in the three-dimensional environment while the first view of the three-dimensional environment is visible. For example, in
In some embodiments, displaying the first user interface object at the first position in the three-dimensional environment in response to detecting the fifth gaze input directed to the second region in the first view of the three-dimensional environment includes: while the fifth gaze input is directed to the second region, displaying the first user interface object with a first size (e.g., an initial size or an intermediate size); and after displaying the first user interface object with the first size and while the fifth gaze input is directed to the second region, displaying the first user interface object with a second size (e.g., an intermediate size or a final, steady state size) that is larger than the first size. In some embodiments, the first user interface object is displayed at the first size when the fifth gaze input is maintained in the second region for at least a first threshold amount of time, and is displayed with increasing sizes after the fifth gaze input is maintained in the second region for more than the first threshold amount of time, until a second threshold amount of time is reached and the first user interface object is displayed with a steady state size. In some embodiments, if the user’s attention ceases to be directed to the second region before the first threshold amount of time is reached, the first computer system forgoes displaying the first user interface object. In some embodiments, if the user’s attention ceases to be directed to the second region before the second threshold amount of time is reached, the first computer system displays the first user interface object and then ceases to display the first user interface object (e.g., forgoes displaying the first user interface object with the steady state size). In some embodiments, the first computer system displays an animated transition between the first size and the second size (e.g., an animated transition of the first user interface object expanding from the first size to the second size) in accordance with the amount of time that the fifth gaze input has remained in the second region. For example, in
In some embodiments, while a respective view of the three-dimensional environment is visible and while the first user interface object is not displayed in the respective view of the three-dimensional environment, the computer system detects, via the one or more input devices, a sixth gaze input directed to a respective region in the respective view of the three-dimensional environment. In response to detecting the sixth gaze input directed to the respective region in the respective view of the three-dimensional environment: in accordance with a determination that the respective view is the first view corresponding to the first viewpoint of the user, and that the respective region has the first spatial relationship to the first view of the three-dimensional environment, the computer system displays the first user interface object in the first view of the three-dimensional environment; and in accordance with a determination that the respective view is the second view corresponding to the second viewpoint of the user, and that the respective region has the first spatial relationship to the second view of the three-dimensional environment, the computer system displays the first user interface object in the second view of the three-dimensional environment. In some embodiments, in response to detecting the sixth gaze input directed to the respective region in the respective view of the three-dimensional environment, in accordance with a determination that the respective region does not have the first spatial relationship to the respective view of the three-dimensional environment, forgoing displaying the first user interface object in the respective view of the three-dimensional environment (e.g., irrespective of whether the respective view is the first view, the second view, or another view of the three-dimensional environment that corresponds to another viewpoint of the user). In some embodiments, the respective region is viewpoint locked and has a fixed spatial relationship to the current field of view provided via the first display generation component, as the respective view of the three-dimensional environment that is visible in the field of view of the first display generation is updated in accordance with the movement of the current viewpoint of the user. For example, as described with reference to
In some embodiments, during a first period of time (e.g., a period of time immediately after a system reset, an initialization, a restart, and/or one or more other state changes of the first computer system) before the first user interface object is displayed in response to a user input that corresponds to a request to display the first user interface object (e.g., the first gaze input that is directed to a respective region of the currently displayed view of the three-dimensional environment that has the first spatial relationship to the currently displayed view of the three-dimensional environment, where the first gaze input meets stability criteria and duration criteria, and/or is detected in conjunction with a first air gesture (e.g., an air pinch gesture, an air tap gesture, or another type of gesture)): the computer system persistently displays (e.g., automatically, without requiring an explicit request via a user input) the first user interface in one or more views of the three-dimensional environment (e.g., while the same view is maintained, or while the view is updated through a series of changes of the viewpoint); while persistently displaying (e.g., automatically, without requiring an explicit request via a user input) the first user interface in one or more views of the three-dimensional environment (e.g., while the same view is maintained, or while the view is updated through a series of changes of the viewpoint), the computer system detects a sixth gaze input directed to the first user interface object; and in response to detecting the sixth gaze input directed to the first user interface object while the first user interface object is persistently displayed in the one or more views of the three-dimensional environment, the computer system ceases to display the first user interface object (e.g., after a threshold amount of time, such as 1 second, 5 seconds, 10 seconds, or 30 seconds, that the sixth gaze input is maintained on the first user interface object). In some embodiments, the first user interface object remains displayed for an extended period of time when the first computer system starts and the first user interface object is displayed for the first time since the first computer system starts or is first initialized, and the first user interface object is dismissed when the first computer system detects the user looks at the first user interface object for the first time. After that, the first computer system displays the first user interface object in response to the user’s request (e.g., via a gaze input directed to the first region of the currently displayed view of the three-dimensional environment) and ceases to display the first user interface object in response to detecting that the user’s attention is no longer directed to first user interface object and/or the plurality of affordances for accessing system functions of the first computer system. For example, as described with reference to
In some embodiments, in response to detecting the sixth gaze input directed to the first user interface object while the first user interface object is persistently displayed in the one or more views of the three-dimensional environment, the computer system displays visual guidance regarding how to redisplay the first user interface object (e.g., after the first user interface object ceases to be displayed in response to the second gaze input) in a respective view of the three-dimensional environment. For example, as described with reference to
In some embodiments, while displaying the visual guidance, the computer system detects that user attention was directed to the visual guidance (e.g., for a threshold amount of time, such as 5 seconds, 10 seconds, 15 seconds, or 30 seconds) before moving away from the visual guidance, wherein the first computer system ceases to display the first user interface object in response to detecting that the user attention has moved outside of the respective region that has the first spatial relationship to the respective view of the three-dimensional environment (e.g., the first view, the second view, or another view of the three-dimensional environment that corresponds to the current viewpoint of the user) (e.g., in accordance with a determination that the user’s gaze has moved away from the respective region for triggering display of the first user interface object, and optionally, has stayed outside of the respective region for at least a threshold amount of time without returning to the respective region). In some embodiments, the first computer system also ceases to display the visual guidance, e.g., in response to detecting that the user attention has moved outside of the respective region that has the first spatial relationship to the respective view of the three-dimensional environment (e.g., before, along with, or after ceasing to display the first user interface object). For example, as described with reference to
In some embodiments, while detecting a respective gaze input directed to the first user interface object (e.g., the first gaze input that is directed to the first user object after the first user interface object is displayed in response to an earlier gaze input on the second region of the first view, or a gaze input that is directed to the second region (e.g., the fifth gaze input mentioned earlier) and that triggers the display of the first user interface object), the computer system increases a size of the first user interface object (e.g., from the final, steady state size to an even larger size, to indicate progress toward displaying the plurality of affordances for accessing system functions). In accordance with a determination that the respective gaze input meets the attention criteria with respect to the first user interface object (e.g., the first gaze input that is detected on the first user interface object for at least a third threshold amount of time, or the fifth gaze input has moved onto the first user interface object after triggering display of the first user interface object, and has been maintained on the first user interface object for at least a fourth threshold amount of time (e.g., the time threshold for triggering display of the system function menu 7024 of
In some embodiments, the first computer system determines that the first gaze input satisfies the attention criteria in accordance with a determination that the first gaze input is maintained on the first user interface object for at least a first amount of time (e.g., an amount of time set by a dwell timer that is started when the first gaze input is detected on the first user interface object), or in accordance with a determination that a first gesture is detected in conjunction with the first gaze input maintained on the first user interface object, before the first gaze input has been maintained on the first user interface object for at least the first amount of time. For example, in some embodiments, if an air pinch gesture is detected before the first gaze input has been maintained on the first user interface object for the first amount of time, the first computer system displays the plurality of affordances right away, without waiting until the first amount of time has been reached. For example, as described with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 8000 in some circumstances has a different appearance as described in the methods 9000-16000 below, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-16000 below). For brevity, these details are not repeated here.
Conditionally displaying content associated with the first notification or displaying the plurality of affordances for performing system operations associated with the first computer system based on whether the first notification occurred recently enough to meet timing criteria causes the computer system to automatically provide access to additional control options that are more likely to be relevant to a current context of the first computer system without displaying additional controls (e.g., controls that are less likely to be relevant to the current context and/or separate controls for displaying the content associated with the first notification, and for displaying the plurality of affordances for performing system operations associated with the first computer system).
The computer system displays, via the first display generation component, a first user interface object (e.g., indicator 7010 of system function menu (
While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via the one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the first gaze input directed to the first user interface object: in accordance with a determination that a first event for a first notification satisfies timing criteria (e.g., the user directs his or her gaze to the first user interface object within a threshold amount of time after the first event for the first notification) (e.g., as described herein with reference to
In some embodiments, while (e.g., after initially) displaying the content associated with the first notification (e.g., in response to detecting that the user’s gaze continues to be directed to the first user interface object), the computer system displays the plurality of affordances for performing system operations associated with the first computer system (e.g., concurrently with the content associated with the first notification). In some embodiments, the plurality of affordances is not displayed (e.g., initially) in response to the first gaze input directed to the first user interface object if the first event for the first notification satisfies the timing criteria, and the plurality of affordances is displayed in accordance with a determination that the user’s gaze continues to be directed to the first user interface object (and/or in response to detecting the user’s continued gaze directed to the first user interface object). This is described above, for example, in the description of
In some embodiments, in accordance with the determination that the first event for the first notification satisfies the timing criteria (e.g., the user directs his or her gaze to the first user interface object within a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) after the first event for the first notification), the first user interface object is displayed with a first appearance that indicates an application associated with the first notification (e.g., the first user interface object is or includes an icon of the associated application); and in accordance with the determination that the first event for the first notification does not satisfy the timing criteria, the first user interface object is displayed with a second appearance, different from the first appearance (e.g., the first user interface is a default indicator (e.g., of system function menu) that does not indicate the application associated with the first notification, and/or the first user interface has another appearance that does not indicate the application associated with the first notification). This is described above, for example, in the description of
In some embodiments, in accordance with the determination that the first event for the first notification satisfies the timing criteria, the computer system ceases to display the first user interface object with the first appearance after a threshold amount of time has elapsed (e.g., since occurrence, generation, or detection of the first event for the first notification) (e.g., and subsequently redisplays the first user interface object with the second appearance after the threshold amount of time has elapsed). This is described above, for example, in the description of
In some embodiments, while displaying the plurality of affordances for performing system operations associated with the first computer system, the computer system detects, via the one or more input devices, an input that corresponds to movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment (e.g., the first view of the three-dimensional environment is visible from the first viewpoint, and a second view of the three-dimensional environment is visible from the second viewpoint, the second view being different from the first view in accordance with the second viewpoint being different from the first viewpoint). In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment (e.g., and while a second view of the three-dimensional environment is visible), the computer system moves the first user interface object by a first amount in the three-dimensional environment, and in accordance with the movement of the viewpoint of the user (e.g., in accordance with a determination that the movement of the viewpoint is detected while displaying the first user interface object), and in accordance with a determination that the movement of the viewpoint is detected while displaying the plurality of affordances for performing system operations associated with the first computer system, the computer system moves the plurality of affordances for performing system operations associated with the first computer system by a second amount in the three-dimensional environment, and in accordance with movement of the viewpoint of the user, wherein the second amount is different from the first amount. For example, if the second amount is less than the first amount, then the first user interface follows the movement of the viewpoint of the user more closely than the plurality of affordances (e.g., during the movement of the viewpoint of the user). Alternatively, if the second amount is greater than the first amount, then the first user interface follows the movement of the viewpoint of the user less closely than the plurality of affordances (e.g., during the movement of the viewpoint of the user).
In some embodiments, the first user interface object and/or the plurality of affordances for performing system operations associated with the first computer system do not move if the movement of the viewpoint of the user does not exceed a threshold amount (e.g., 5 degrees, 10 degrees, 25 degrees, 45 degrees, 90 degrees, 120 degrees, or any threshold angle between 0 and 120 degrees, and/or distance of 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters) of movement (e.g., small, inadvertent movement or sway of the user’s hand(s) while holding the first computer system or of the user’s head while wearing a head-mounted display of the first computer system), whereas the first user interface object and/or the plurality of affordance for performing system operations associated with the first computer system move if the movement of the viewpoint of the user exceeds the threshold amount of movement (e.g., intentional movement to change the viewpoint of the user). In some embodiments, in accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the first user interface object at the first position in the three-dimensional environment while a second view of the three-dimensional environment is visible (e.g., even though the first user interface object no longer has the first spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the first user interface object at a respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
In some embodiments, the plurality of affordances for performing system operations associated with the first computer system is displayed at a second position in the three-dimensional environment (e.g., in response to detecting the first gaze input directed to the first user interface object and optionally in accordance with the determination that the first event for the first notification does not satisfy the timing criteria), wherein the second position in the three-dimensional environment has a second spatial arrangement relative to the respective portion of the user. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the plurality of affordances for performing system operations associated with the first computer system at the second position in the three-dimensional environment while the second view of the three-dimensional environment is visible (e.g., even though the plurality of affordances for performing system operations associated with the first computer system no longer have the second spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the plurality of affordances for performing system operations associated with the first computer system at a respective position in the three-dimensional environment having the second spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
This is described above, for example, in the description of
In some embodiments, the computer system detects, via the one or more input devices, an input that corresponds to movement of the viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, and in accordance with the determination that the first event for the first notification satisfies the timing criteria (e.g., the user directs his or her gaze to the first user interface object within a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) after the first event for the first notification): the computer system displays the first user interface object with a first appearance that indicates an application associated with the first notification; and the computer system moves the first user interface object by a third amount in the three-dimensional environment, and in accordance with the movement of the viewpoint of the user.
In response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, and in accordance with the determination that the first event for the first notification does not satisfy the timing criteria: the computer system displays the first user interface object with a second appearance, different from the first appearance (e.g., the appearance of the indicator 7010 of system function menu (
In some embodiments, the first user interface object does not move if the movement of the viewpoint of the user does not exceed a threshold amount of movement (e.g., 5 degrees, 10 degrees, 25 degrees, 45 degrees, 90 degrees, 120 degrees, or any threshold angle between 0 and 120 degrees, and/or distance of 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters) (e.g., small, inadvertent movement or sway of the user’s hand(s) while holding the first computer system or of the user’s head while wearing a head-mounted display of the first computer system), whereas the first user interface object moves if the movement of the viewpoint of the user exceeds the threshold amount of movement (e.g., intentional movement to change the viewpoint of the user). In some embodiments, in accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the first user interface object at the first position in the three-dimensional environment while a second view of the three-dimensional environment is visible. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the first user interface object at a respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
This is described above, for example, in the description of
In some embodiments, the computer system detects, via the one or more input devices, an input that corresponds to movement of the viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment: in accordance with a determination that the movement of the viewpoint is detected while displaying the plurality of affordances for performing system operations associated with the first computer system, the computer system moves the plurality of affordances for performing system operations associated with the first computer system by a fifth amount in the three-dimensional environment, and in accordance with the movement of the viewpoint of the user; and in accordance with a determination that the movement of the viewpoint is detected while displaying the content associated with the first notification, the computer system moves the content associated with the first notification by a sixth amount in the three-dimensional environment in accordance with the movement of the viewpoint of the user, wherein the sixth amount is different from the fifth amount. For example, if the sixth amount is less than the fifth amount, then the plurality of affordances follows the movement of the viewpoint of the user more closely than the content associated with the first notification (e.g., for a given amount of movement of the viewpoint of the user, the notification content would move less from a prior position toward a new position corresponding to the current viewpoint of the user, and thus would follow the viewpoint of the user more slowly (less closely) than would the plurality of affordances). Alternatively, if the sixth amount is greater than the fifth amount, then the plurality of affordances follows the movement of the viewpoint of the user less closely than the content associated with the first notification (e.g., for a given amount of movement of the viewpoint of the user, the notification content would move more from a prior position toward a new position corresponding to the current viewpoint of the user, and thus would follow the viewpoint of the user faster (more closely) than would the plurality of affordances). This is described above, for example, in the description of
In some embodiments, the plurality of affordances for performing system operations associated with the first computer system are concurrently displayed with the content associated with the first notifications. In some embodiments, the plurality of affordances for performing system operations associated with the first computer system and/or the content associated with the first notification do not move if the movement of the viewpoint of the user does not exceed a threshold amount of movement (e.g., 5 degrees, 10 degrees, 25 degrees, 45 degrees, 90 degrees, 120 degrees, or any threshold angle between 0 and 120 degrees, and/or distance of 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters) (e.g., small, inadvertent movement or sway of the user’s hand(s) while holding the first computer system or of the user’s head while wearing a head-mounted display of the first computer system), whereas the plurality of affordance for performing system operations associated with the first computer system and/or the content associated with the first notification move if the movement of the viewpoint of the user exceeds the threshold amount of movement (e.g., intentional movement to change the viewpoint of the user).
In some embodiments, the plurality of affordances for performing system operations associated with the first computer system is displayed at a second position in the three-dimensional environment (e.g., in response to detecting the first gaze input directed to the first user interface object and optionally in accordance with the determination that the first event for the first notification does not satisfy the timing criteria), wherein the second position in the three-dimensional environment has a second spatial arrangement relative to the respective portion of the user. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the plurality of affordances for performing system operations associated with the first computer system at the second position in the three-dimensional environment while the second view of the three-dimensional environment is visible (e.g., even though the plurality of affordances for performing system operations associated with the first computer system no longer have the second spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the plurality of affordances for performing system operations associated with the first computer system at a respective position in the three-dimensional environment having the second spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
In some embodiments, the content associated with the first notification is displayed at a third position in the three-dimensional environment, wherein the third position in the three-dimensional environment has a third spatial arrangement relative to the respective portion of the user. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the content associated with the first notification at the third position in the three-dimensional environment while the second view of the three-dimensional environment is visible (e.g., even though the content associated with the first notification no longer has the third spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the content associated with the first notification at a respective position in the three-dimensional environment having the third spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
Moving the plurality of affordances for performing system operations associated with the first computer system by a different amount in the three-dimensional environment than the content associated with the first notification, in accordance with a given amount of movement of the viewpoint of the user, provides improved visual feedback distinguishing different user interface elements and enables visually emphasizing some user interface elements (e.g., that are displayed as following the user’s viewpoint more closely) over others (e.g., that are displayed as following the user’s viewpoint less closely).
In some embodiments, the computer system maintains display of the first user interface object in the first view of the three-dimensional environment (e.g., while displaying the content associated with the first notification without displaying the plurality of affordances for performing system operations associated with the first computer system). This is shown in
In some embodiments, while displaying the content associated with the first notification (e.g., without displaying the plurality of affordances for performing system operations associated with the first computer system), the computer system detects, via the one or more input devices, a second gaze input directed to the first user interface object. In some embodiments, in response to detecting the second gaze input directed to the first user interface object, the computer system optionally ceases to display the content associated with the first notification, and the computer system displays the plurality of affordances (e.g., the plurality of affordances described herein with reference to
In some embodiments, while displaying the plurality of affordances for performing system operations associated with the first computer system (e.g., without displaying the content associated with the first notification), the computer system detects, via the one or more input devices, a third gaze input directed to the first user interface object. In some embodiments, in response to detecting the third gaze input directed to the first user interface object, the computer system optionally ceases to display the plurality of affordances for performing system operations associated with the first computer system (e.g., a volume control, a search affordance, a notification center, a control panel, and/or a virtual assistant), and the computer system displays the content associated with the first notification. In some embodiments, ceasing to display the plurality of affordances and instead displaying the content associated with the first notification are performed in accordance with a determination that the first event for the first notification (e.g., still) satisfies the timing criteria. This is described above, for example, in the description of
In some embodiments, in response to detecting the first gaze input directed to the first user interface object, and in accordance with a determination that the first computer system is in an active communication session, the computer system displays a session user interface at a location with a respective relationship to (e.g., directly adjacent to, above, below, within a predefined threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of the plurality of affordances for performing system operations associated with the first computer system (e.g., in accordance with a determination that the plurality of affordances is displayed). In some embodiments, in accordance with the determination that the first event for the first notification satisfies the timing criteria and the determination that the first computer system is in the active communication session, the session user interface is displayed at a location with a respective relationship to the content associated with the first notification.
In some embodiments, the session user interface includes a plurality of affordances corresponding to the active communication session. For example, the plurality of affordances include affordances for: leaving the active communication session, displaying past communication information (e.g., previously received text messages) corresponding to one or more other users in the active communication session, displaying information about the active communication session (e.g., a list of users in the active communication session), muting or unmuting a microphone of the first computer system, displaying previously shared content (e.g., shared by and/or with one or more other users in the active communication session), displaying recently shared content (e.g., shared by and/or with one or more other users in the active communication session), and/or sharing content (e.g., photos, photo albums, Internet content, and/or a screen of the first computer system) with one or more other users in the active communication session.
This is shown in
In some embodiments, while concurrently displaying the session user interface and the plurality of affordances for performing system operations associated with the first computer system, and in accordance with a determination that the user’s gaze is directed to the plurality of affordances for performing system operations associated with the first computer system, the computer system visually deemphasizes the session user interface (e.g., relative to the plurality of affordances for performing system operations associated with the first computer system). This is described above, for example, in the description of
In some embodiments, while concurrently displaying the session user interface and the plurality of affordances for performing system operations associated with the first computer system, and in accordance with a determination that the user’s gaze is directed to the session user interface, the computer system visually deemphasizes (e.g., blurs, fades, and/or shrinks) the plurality of affordances for performing system operations associated with the first computer system (e.g., relative to the session user interface). This is described above, for example, in the description of
In some embodiments, while displaying the content associated with the first notification, the computer system detects, via the one or more input devices, a first input directed to the content associated with the first notification. In some embodiments, in response to detecting the first input directed to the content associated with the first notification, the computer system launches a first application associated with the first notification, which including displaying, in the three-dimensional environment, a user interface of the first application that is associated with first notification. For example, the first notification is a notification of a received communication (e.g., email or text message), and launching the first application includes displaying a user interface of the first application with a view of the received communication (e.g., displaying the received email in an email application user interface, or displaying the received text message in the context of a conversation in a messages application user interface). In another example, the first notification is a reminder for a scheduled calendar event, and launching the first application includes displaying the calendar event information in a calendar application user interface. This is shown in
In some embodiments, the computer system detects that the first input meets application launch criteria, and the application launch criteria require the first input include an air pinch and drag gesture, in order for the application launch criteria to be met. In some embodiments, the application launch criteria require that the air pinch and drag gesture be performed while the user’s gaze is directed to the first notification or to the content associated with the first notification). In some embodiments, the first input directed to the content associated with the first notification (e.g., in response to which the first application is launched) includes an air pinch and drag gesture, and is detected while the user’s attention (e.g., gaze) is directed to the content associated with the first notification. As described herein with reference to
In some embodiments, the computer system detects, via the one or more input devices, an input that corresponds to movement of the viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment: in accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint is detected while displaying the content associated with the first notification, the computer system moves the content associated with the first notification by a seventh amount in the three-dimensional environment, and in accordance with the movement of the viewpoint of the user; and in accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint is detected while displaying the user interface of the first application that is associated with the first notification, the computer system moves the user interface of the first application that is associated with the first notification by an eighth amount in the three-dimensional environment, and in accordance with the movement of the viewpoint of the user, wherein the eighth amount is different from the seventh amount. For example, if the eighth amount is less than the seventh amount, then the content associated with the first notification follows the movement of the viewpoint of the user more closely than the user interface of the first application that is associated with the first notification (e.g., for a given amount of movement of the viewpoint of the user, the application user interface would move less from a prior position toward a new position corresponding to the current viewpoint of the user, and thus would follow the viewpoint of the user more slowly (less closely) than would the notification content). Alternatively, if the eighth amount is greater than the seventh amount, then the content associated with the first notification follows the movement of the viewpoint of the user less closely than the user interface of the first application that is associated with the first notification (e.g., for a given amount of movement of the viewpoint of the user, the application user interface would move more from a prior position toward a new position corresponding to the current viewpoint of the user, and thus would follow the viewpoint of the user faster (more closely) than would the notification content). This is described above, for example, in the description of
In some embodiments, the plurality of affordances for performing system operations associated with the first computer system and/or the content associated with the first notification do not move if the movement of the viewpoint of the user does not exceed a threshold amount of movement (e.g., 5 degrees, 10 degrees, 25 degrees, 45 degrees, 90 degrees, 120 degrees, or any threshold angle between 0 and 120 degrees, and/or distance of 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 1 meter, 5 meters, or any threshold distance between 0 and 5 meters) (e.g., small, inadvertent movement or sway of the user’s hand(s) while holding the first computer system or of the user’s head while wearing a head-mounted display of the first computer system), whereas the plurality of affordance for performing system operations associated with the first computer system and/or the content associated with the first notification move if the movement of the viewpoint of the user exceeds the threshold amount of movement (e.g., intentional movement to change the viewpoint of the user).
In some embodiments, the content associated with the first notification is displayed at a third position in the three-dimensional environment, wherein the third position in the three-dimensional environment has a third spatial arrangement relative to the respective portion of the user. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the content associated with the first notification at the third position in the three-dimensional environment while the second view of the three-dimensional environment is visible (e.g., even though the content associated with the first notification no longer has the third spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the content associated with the first notification at a respective position in the three-dimensional environment having the third spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
In some embodiments, the first application associated with the first notification is displayed at a fourth position in the three-dimensional environment, wherein the fourth position in the three-dimensional environment has a fourth spatial arrangement relative to the respective portion of the user. In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint does not satisfy the threshold amount (e.g., angle and/or distance) of movement, the computer system maintains display of the first application associated with the first notification at the fourth position in the three-dimensional environment while the second view of the three-dimensional environment is visible (e.g., even though the first application associated with the first notification no longer has the fourth spatial arrangement relative to the respective portion of the user). In accordance with a determination that the movement of the viewpoint of the user from the first viewpoint to the second viewpoint satisfies the threshold amount of movement, the computer system maintains display of the first application associated with the first notification at a respective position in the three-dimensional environment having the fourth spatial arrangement relative to the respective portion of the user while the second view of the three-dimensional environment is visible.
Moving the content associated with the first notification by a different amount in the three-dimensional environment than the user interface of the first application that is associated with the first notification, in accordance with a given amount of movement of the viewpoint of the user, provides improved visual feedback distinguishing different user interface elements and enables visually emphasizing some user interface elements (e.g., that are displayed as following the user’s viewpoint more closely) over others (e.g., that are displayed as following the user’s viewpoint less closely).
In some embodiments, while displaying the content associated with the first notification, the computer system detects, via the one or more input devices, a second input that is directed to the content associated with the first notification and that meets dismissal criteria, wherein the dismissal criteria require the second input include an air pinch and drag gesture (optionally in conjunction with a gaze input directed to the content associated with the first notification), in order for the dismissal criteria to be met. In some embodiments, in response to detecting the second input, the computer system ceases to display the content associated with the first notification. In some embodiments, if the first user interface object was displayed with an appearance that indicates an application associated with the first notification prior to (and/or while) displaying the content associated with the first notification, the first user interface object is redisplayed with an appearance that does not indicate the application associated with the first notification in response to detecting the second input (e.g., to dismiss the content associated with the first notification). In some embodiments, while displaying the content associated with the first notification, and while the user’s attention (e.g., gaze) is directed to the content associated with the first notification, the computer system detects an air pinch and drag gesture, in response to which the computer system ceases to display the content associated with the first notification. This is described herein with reference to
In some embodiments, while displaying the content associated with the first notification, the computer system detects, via the one or more input devices, a third input (e.g., in conjunction with a gaze input directed to the content associated with the first notification) directed to the content associated with the first notification. In some embodiments, in response to detecting the third input directed to the content associated with the first notification: in accordance with a determination that the third input includes a first air gesture and movement of the user’s hand in a third direction (e.g., the third direction is the same as the first direction) during the first air gesture, the computer system launches a first application associated with the first notification (e.g., displaying a user interface of the first application that is associated with the first notification); and in accordance with a determination that the third input includes a second air gesture and, movement of the user’s hand in a fourth direction, different from the third direction (e.g., the fourth direction is the same as the second direction) during the second air gesture, the computer system ceases to display the content associated with the first notification (e.g., without launching the first application, and/or without displaying a user interface of the first application that is associated with the first notification). This is described herein with reference to
In some embodiments, in response to detecting the first gaze input directed to the first user interface object, and in accordance with the determination that the first event for the first notification satisfies the timing criteria, the computer system displays the plurality of affordances for performing system operations associated with the first computer system (e.g., the plurality of affordances for performing system operations associated with the first computer system described herein with reference to
In some embodiments, in response to detecting the first gaze input directed to the first user interface object, and in accordance with the determination that the first event for the first notification satisfies the timing criteria, the content associated with the first notification is displayed without displaying the plurality of affordances for performing system operations associated with the first computer system. This is shown in
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator, sometimes appearing as a notification indicator) in the method 9000 has characteristics of the first user interface object (e.g., system control indicator) in the methods 8000 and 10000-16000, and the user interface elements that are displayed (e.g., the additional content associated with a notification) may be replaced by, or concurrently displayed with, other user interface elements (e.g., the plurality of affordances for performing system operations associated with the first computer system, a user interface that includes an affordance for joining a communication session, and/or the user interface elements of the methods 8000, 10000-16000). For brevity, these details are not repeated here.
Conditionally displaying a user interface that includes an affordance for joining a communication session and/or displaying the plurality of affordances for performing system operations associated with the first computer system, based on whether a request to join the communication session is active and meets timing criteria, enables access to additional control options relevant to a current context of the first computer system without displaying additional controls (e.g., separate controls for displaying the first user interface that includes an affordance for joining the first communication session, and for displaying the plurality of affordances for performing system operations associated with the first computer system).
The computer system displays, via the first display generation component, a first user interface object (e.g., indicator 7010 of system function menu (
While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects, via the one or more input devices, a first gaze input directed to the first user interface object. In response to detecting the gaze input directed to the first user interface object: in accordance with a determination that there is a request for the first computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time before the user directs his or her attention to the first user interface object), the computer system displays a first user interface that includes an affordance for joining the first communication session (e.g., concurrently with display of the first user interface object) (e.g., incoming call user interface 7068 (
In some embodiments, in accordance with a determination that there is a request for the first computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies the timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the computer system displays the first user interface object with a first appearance (e.g., including one or more of a first color, a first shape, a user’s avatar, and/or an application icon); and in accordance with a determination that there is not a request to join a communication session that satisfies the timing criteria (e.g., in accordance with a determination that the computer system has not received any requests for the computer to join a communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the computer system displays the first user interface object with a second appearance (e.g., different from the first appearance). For example, the user interface object 7064 in
In some embodiments, displaying the first user interface object with the first appearance includes displaying the first user interface object with a first color, and displaying the first user interface object with the second appearance includes displaying the first user interface object with a second color different from the first color. This is described above, for example, in the description of
In some embodiments, displaying the first user interface object with the first appearance includes displaying an animation of the first user interface object (e.g., an animation involving pulsing a border of the first user interface object, an animation that involves changing a color of the first user interface object, and/or an animation involving changing a shape of the first user interface object). This is described above, for example, in the description of
In some embodiments, in accordance with the determination that there is a request for the first computer system to join the first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies the timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the computer system displays the animation of the first user interface object without generating audio output (e.g., in conjunction with displaying the first user interface object with the first appearance). This is described above, for example, in the description of
In some embodiments, in accordance with a determination that there is a request for the first computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies the timing criteria (e.g., in accordance with a determination that the first computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the computer system displays a visual indication (e.g., a name, an avatar, and/or a profile picture) of a user associated with the request for the computer system to join the first communication session (e.g., the user that sent the request for the computer system to join the first communication session). In some embodiments, in accordance with a determination that there is a request for the computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the visual indication is displayed as a portion of (or separate from, but within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of) the first user interface object. In some embodiments, the visual indication is displayed as a portion of the first user interface (e.g., in addition to being displayed as a portion of the first user interface object). This is described above, for example, in the description of
In some embodiments, the computer system displays the visual indication (e.g., a name, an avatar, and/or a profile picture) of the user associated with the request for the first computer system to join the first communication session before detecting the first gaze input directed to the first user interface object that satisfies the timing criteria (e.g., the user that sent the request for the computer system to join the first communication session). In some embodiments, in accordance with a determination that there is a request for the computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the visual indication is displayed as a portion of (or separate from, but within a threshold distance (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 and 5 cm) of) the first user interface object. In some embodiments, the visual indication is displayed as a portion of the first user interface (e.g., in addition to being displayed as a portion of the first user interface object). In some embodiments, the visual indication of the user associated with the request for the computer system to join the first communication session is displayed for a threshold amount of time (e.g., and the computer system ceases to display the visual indication of the user associated with the request for the computer system to join the first communication session after the threshold amount of time).
After (e.g., while) displaying the first user interface that includes an affordance for joining the first communication session (e.g., concurrently with display of the first user interface object), the computer system detects that the user’s gaze is no longer directed to the first user interface object (e.g., and not directed to the first user interface). In some embodiments, in response to detecting that the user’s gaze is no longer directed to the first user interface object, the computer system displays the first user interface object with a third appearance (e.g., instead of the first appearance) (e.g., to provide visual feedback that the first user interface that includes an affordance for joining the first communication session was already displayed). In some embodiments, the third appearance is the same as the second appearance.
This is described above, for example, in the description of
In some embodiments, while displaying the first user interface object at the first position in the three-dimensional environment that has the first spatial arrangement relative to the respective portion of the user, the computer system detects, via the one or more input devices, an input that corresponds to movement of a viewpoint of the user from a first viewpoint (e.g., from which the first view of the three-dimensional environment is visible) to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system maintains display of the first user interface object at a respective position in the three-dimensional environment having the first spatial arrangement relative to the respective portion of the user while a second view of the three-dimensional environment is visible (e.g., the second view is visible from the second viewpoint, and the second view is different from the first view). In some embodiments, the first user interface object is maintained at a respective position in the three-dimensional environment having the first spatial arrangement relative to the viewpoint of the user. In some embodiments, the first user interface object is displayed at one or more successive positions in the three-dimensional environment as the viewpoint of the user moves such that one or more successive corresponding views of the three-dimensional environment become visible, each of the successive positions of the first user interface object having the first spatial arrangement relative to the respective portion of the user. In some embodiments, the first user interface object is maintained at the first location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment. This is described above, for example, in the description of
In some embodiments, in response to detecting the first gaze input directed to the first user interface object, in accordance with a determination that there is a request for the first computer system to join a first communication session (e.g., a phone call, a video call, a shared computer-generated experience, and/or a shared virtual or augmented reality environment) that satisfies the timing criteria (e.g., in accordance with a determination that the computer system has received a request for the computer system to join a first communication session less than a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) before the user directs his or her attention to the first user interface object), the computer system concurrently displays the first user interface that includes an affordance for joining the first communication session (e.g., concurrently with display of the first user interface object) and the plurality of affordances for performing system operations associated with the first computer system (e.g., a volume control, a search affordance, a notification center, a control panel, and/or a virtual assistant). This is shown in
In some embodiments, the first communication session is an augmented reality or virtual reality communication session (e.g., an extended reality (XR) communication session). This is shown in
In some embodiments, the request for the first computer system to join the first communication session is received from a second user (e.g., different from a user of the computer system), and the first user interface is displayed in accordance with a determination that the first gaze input directed to the first user interface object is detected within a threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) of receiving the request for the first computer system to join the first communication session (e.g., in addition to the determination that the first event for the first notification satisfies the timing criteria). In accordance with a determination (e.g., made while displaying the first user interface) that the affordance for joining the first communication session is not activated within the threshold amount of time of receiving the request (e.g., the request for the computer system to join the first communication session expires (e.g., after a threshold amount of time)): the computer system ceases to display the first user interface that includes the affordance for joining the first communication session; the computer system displays a second user interface, the second user interface including a visual indication that the affordance for joining the first communication session was not activated within the threshold amount of time; and the computer system displays (e.g., as a portion of the second user interface, or otherwise concurrently with the second user interface) an affordance for initiating a second communication session with the second user (e.g., an affordance for sending, to the second user that initiated the request for the computer system to join the first communication session, a request to join a second communication session). This is shown in
In some embodiments, the computer system displays the second user interface (e.g., a missed call user interface) for a first threshold amount of time and ceases to display the second user interface after the first threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) has elapsed. This is shown in
In some embodiments, in accordance with a determination that the first communication session is an augmented reality or virtual reality communication session (e.g., is initiated from another computer system that supports augmented reality and/or virtual reality communication sessions), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a second threshold amount of time (e.g., 30 seconds, 60 seconds, or any threshold time between 21 and 60 seconds) before detecting the first gaze input directed to the first user interface object. In accordance with a determination that the first communication session is not an augmented reality or virtual reality communication session (e.g., neither an augmented reality nor virtual reality communication session, as in not an extended reality communication session, for example due to being initiated from another computer system that does not support augmented reality nor virtual reality communication sessions), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a third threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, or any threshold time between 0 and 20 seconds) before detecting the first gaze input directed to the first user interface object, wherein the third threshold amount of time is less than the second threshold amount of time.
This is shown in
In some embodiments, in accordance with a determination that the first communication session is an augmented reality or virtual reality communication session, the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a fourth threshold amount of time (e.g., 30 seconds, 60 seconds, or any threshold time between 21 and 60 seconds) before detecting the first gaze input directed to the first user interface object (e.g., the fourth threshold amount of time is the second threshold amount of time described above for requests for communication sessions that are augmented reality or virtual reality communication sessions). In accordance with a determination that the first communication session is a video or audio communication session (e.g., a video communication session request sent using a videotelephony protocol, whether the request is for a video call or an audio-only call with video disabled or turned off yet still supported by the videotelephony protocol), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a fifth threshold amount of time (e.g., 5 seconds, 10 seconds, 20 seconds, or any threshold time between 0 and 20 seconds) before detecting the first gaze input directed to the first user interface object, wherein the fifth threshold amount of time is less than the fourth threshold amount of time (e.g., the fifth threshold amount of time is the third threshold amount of time of operation described above for requests for communication sessions that are not augmented reality or virtual reality communication sessions).
This is shown in
In some embodiments, in accordance with a determination that the first communication session is a video communication session (e.g., that uses a videotelephony protocol, optionally without regard to whether video is disabled or turned off if video is supported by the videotelephony protocol (e.g., an audio-only call over the videotelephony protocol is optionally considered a “video communication session” as the term is used herein)), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a sixth threshold amount of time (e.g., 20 seconds, 30 seconds, 60 seconds, or any threshold time between 11 and 60 seconds) before detecting the first gaze input directed to the first user interface object (e.g., the sixth threshold amount of time is the third threshold amount of time and/or the fifth threshold amount of time of operation described above for requests for communication sessions that are augmented reality or virtual reality communication sessions). In accordance with a determination that the first communication session is an audio communication session (e.g., that uses a communication protocol that does not support video, such as a telephone call), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a seventh threshold amount of time (e.g., 5 seconds, 10 seconds, or any threshold time between 0 and 10 seconds) before detecting the first gaze input directed to the first user interface object, wherein the seventh threshold amount of time is less than the sixth threshold amount of time. In some embodiments, in accordance with the determination that the first communication session is an audio communication session (e.g., an audio-only communication session such as a phone call), the first user interface that includes the affordance for joining the first communication session is displayed before detecting the first gaze input directed to the first user interface object (e.g., a gaze input directed to the first user interface object is not required in order for the first user interface to be displayed).
This is shown in
In some embodiments, in accordance with a determination that the first communication session is an audio communication session (e.g., an audio-only communication session such as a phone call), the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than an eighth threshold amount of time (e.g., 5 seconds, 10 seconds, or any threshold time between 0 and 10 seconds) before detecting the first gaze input directed to the first user interface object, and/or the first user interface that includes an affordance for joining the first communication session is displayed with a first level of prominence. In some embodiments, in accordance with a determination that the first communication session is not an audio communication session, the request for the first computer system to join the first communication session satisfies the timing criteria if the request was received less than a ninth threshold amount of time (e.g., 20 seconds, 30 seconds, 60 seconds, or any threshold time between 11 and 60 seconds) before detecting the first gaze input directed to the first user interface object, wherein the ninth threshold amount of time is greater than the eighth threshold amount of time (e.g., a request to join an audio communication expires more quickly than a request to join another type of communication session (e.g., video, virtual reality, and/or augmented reality)), and/or the first user interface that includes an affordance for joining the first communication session is displayed with a second level of prominence, wherein the second level of prominence is less than the first level of prominence (e.g., a request to join an audio communication session is displayed with greater prominence, but for a shorter duration, than requests to join other types of communication sessions). This is shown in
In some embodiments, the first user interface displayed with the second level of prominence is displayed with a smaller size (e.g., as an indicator (e.g., of system function menu) instead of a banner), in a less central portion of the user’s viewpoint (e.g., in an x-direction and/or y-direction), and/or at a position that appears further away (e.g., in a z-direction) from the viewpoint of the user, than when displayed with the first level of prominence. In some embodiments, the first user interface displayed with the second level of prominence is not animated (e.g., and is animated when displayed with the first level of prominence). In some embodiments, the first user interface is less persistent (e.g., easier to dismiss) when displayed with the second level of prominence than when displayed with the first level of prominence. For example, when displayed with the second level of prominence, the first user interface ceases to be displayed if the user’s attention is no longer directed to the first user interface. In contrast, when displayed with the first level of prominence, the first user interface continues to be displayed irrespective of the user’s attention (e.g., even if the user’s attention is not directed to the first user interface). In some embodiments, displaying the first user interface with the second level of prominence includes outputting less prominent audio (e.g., a discrete alert sound or tone), whereas displaying the first user interface with the first level of prominence includes outputting more prominent audio (e.g., continuous and/or repeated ringing).
Setting a time limit for a user to interact with an audio communication session request to be shorter than the time limit for the user to interact with other types of communication session requests and presenting the audio communication session request more prominently than other types of requests provides feedback indicating that audio communication session requests (e.g., phone calls) are more urgent than other types of communication session requests and reduces the number of inputs needed to establish audio communication sessions between users (e.g., by increasing the likelihood that the receiving user will notice the less persistent incoming audio communication session request before the request expires, thereby reducing the need for additional requests to be sent back and forth between users).
In some embodiments, in accordance with a determination that there is an active communication session (e.g., the first communication session, if the user of the first computer system activates the affordance for joining the first communication session), and in accordance with a determination that the user’s gaze is directed to the first user interface object, the computer system displays a third user interface that includes a plurality of affordances for performing operations associated with the active communication session (e.g., leaving the communication session, sending one or more requests to other users to join the communication session, and/or sharing content in the communication session), wherein the third user interface is displayed with a respective spatial arrangement (e.g., directly below, or other required spatial arrangement) relative to the first user interface object. In some embodiments, the third user interface is displayed while (e.g., as long as) the active communication session remains active. In some embodiments, the third user interface is displayed in response to detecting a gaze input directed to the first user interface object while the active communication session is active (e.g., as long as the computer system detects that the user’s gaze is directed to the first user interface object). This is described above, for example, in the description of
In some embodiments, the computer system displays, via the first display generation component, the third user interface while the first view of the three-dimensional environment is visible, wherein the third user interface is displayed at a second position in the three-dimensional environment, and wherein the second position in the three-dimensional environment has a second spatial arrangement relative to the respective portion of the user (e.g., relative to a viewpoint of the user). In some embodiments, the third user interface is displayed at a second location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment. In some embodiments, the 3D environment is a virtual environment, video passthrough (e.g., based on a camera feed), or true passthrough of the physical environment surrounding or in view of the first computer system.
While displaying the third user interface at the second position in the three-dimensional environment that has the second spatial arrangement relative to the respective portion of a user, the computer system detects, via the one or more input devices, an input that corresponds to movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system maintains display of the third user interface at a respective position in the three-dimensional environment having the second spatial arrangement relative to the respective portion of the user while a second view of the three-dimensional environment is visible (e.g., the second view is different from the first view). In some embodiments, the third user interface is maintained at a respective position in the three-dimensional environment having the second spatial arrangement relative to the viewpoint of the user. In some embodiments, the third user interface is maintained at the second location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment.
This is described above, for example, in the description of
In some embodiments, the computer system displays, via the first display generation component, the third user interface at a respective position in the three-dimensional environment that has a second spatial arrangement relative to the first user interface object. While displaying the first user interface object concurrently with the third user interface in the first view of the three-dimensional environment, the computer system detects, via the one or more input devices, input that corresponds to movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system maintains display of the third user interface at a respective position in the three-dimensional environment having the second spatial arrangement relative to the first user interface object while a second view of the three-dimensional environment is visible. This is described above, for example, in the description of
In some embodiments, in accordance with the determination that there is the active communication session, and in accordance with the determination that the user’s gaze is directed to the first user interface object, the computer system displays, via the first display generation component, the plurality of affordances for performing system operations associated with the first computer system while the first view of the three-dimensional environment is visible, wherein the plurality of affordances for performing system operations associated with the first computer system are displayed at a third position in the three-dimensional environment, and wherein the third position in the three-dimensional environment has a third spatial arrangement relative to the third user interface (e.g., relative to a viewpoint of the user). In some embodiments, the plurality of affordances for performing system operations associated with the first computer system is displayed at a third location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment. In some embodiments, the 3D environment is a virtual environment, video passthrough (e.g., based on a camera feed), or true passthrough of the physical environment surrounding or in view of the first computer system.
While displaying the plurality of affordances for performing system operations associated with the first computer system and the third user interface in the first view of the three-dimensional environment, the computer system detects, via the one or more input devices, an input that corresponds to movement of a viewpoint of the user from a first viewpoint to a second viewpoint in the three-dimensional environment. In some embodiments, in response to detecting the input that corresponds to movement of the viewpoint of the user from the first viewpoint to the second viewpoint in the three-dimensional environment, the computer system maintains display of the plurality of affordances for performing system operations associated with the first computer system at a respective position in the three-dimensional environment having the third spatial arrangement relative to the third user interface while a second view of the three-dimensional environment is visible (e.g., the second view is different from the first view). In some embodiments, the plurality of affordances for performing system operations associated with the first computer system is maintained at a respective position in the three-dimensional environment having the third spatial arrangement relative to the viewpoint of the user). In some embodiments, the plurality of affordances for performing system operations associated with the first computer system is maintained at the third location on, or relative to, the first display generation component that is independent of the contents of the first view of the three-dimensional environment.
This is described above, for example, in the description of
In some embodiments, the computer system detects a user input that initiates a request, to a user of a second computer system (e.g., a user who is not currently participating in the active communication session), to join a communication session. While the request to join the communication session is active, and in accordance with a determination that the user’s gaze is directed to the first user interface object, the computer system displays a fourth user interface that includes a plurality of affordances for performing operations associated with the requested communication session (e.g., leaving the communication session (and/or cancelling the request to the other user to join the communication session), sending one or more requests to additional users to join the communication session, and/or sharing content in the communication session) at a respective location (e.g., directly below, or other required location) relative to the first user interface object. In some embodiments, the fourth user interface is displayed while (e.g., as long as) the active communication session remains active. In some embodiments, the fourth user interface is displayed in response to detecting a gaze input directed to the first user interface object while the active communication session is active (e.g., as long as the computer system detects that the user’s gaze is directed to the first user interface object). This is described above, for example, in the description of
In some embodiments, while the request to join the communication session is active, and in accordance with a determination that the user’s gaze is no longer directed to the first user interface object, the computer system ceases to display the fourth user interface, while maintaining display of the first user interface object. This is described above, for example, in the description of
In some embodiments, after ceasing to display the fourth user interface, and while the request to join the communication session is active, the computer system detects, via the one or more input devices, a second gaze input directed to the first user interface object. In some embodiments, in response to detecting the second gaze input directed to the first user interface object, the computer system redisplays the fourth user interface. This is described above, for example, in the description of
In some embodiments, while the request to join the communication session is active (e.g., before the user of the second computer system accepts the request to join the active communication session), the computer system displays an animation of the first user interface object (e.g., an animation that includes: the first user interface object changing color, the first user interface object changing shape, the first user interface object changing size, a border of the first user interface object pulsing (e.g., without changing a size of the first user interface object itself), the first user interface object fading in and out, and/or the first user interface object moving (e.g., up and down, side to side, and/or rotating)). This is described above, for example, in the description of
In some embodiments, the computer system detects that the second computer system has joined the requested communication session. In some embodiments, in response to detecting that the second computer system has joined the requested communication session, the first computer system displays a visual representation of the user of the second computer system (e.g., an avatar of the user of the second computer system, and/or a profile picture of the user of the second computer system). This is described above, for example, in the description of
In some embodiments, the plurality of affordances for performing operations associated with the communication session (e.g., an active communication session, or a requested communication session, for which the plurality of affordances for performing communication session operations is displayed) include an affordance for toggling display of a visual representation (e.g., an avatar, a profile picture, a name, and/or initials) of a user (e.g., a user of a computer system other than the first computer system, and/or the user of the first computer system) in the communication session. The computer system detects an input directed to the affordance for toggling display of the visual representation of the user. In some embodiments, in response to detecting the input directed to the affordance for toggling display of the visual representation of the user: in accordance with a determination that the input directed to the affordance is detected while a first visual representation (e.g., a realistic likeness, such as a three-dimensional character, a two-dimensional image, an icon, and/or an avatar that is based on the user’s likeness) of a respective user is displayed, the computer system replaces display of the first visual representation of the respective user with a second visual representation (e.g., an abstract representation that is not based on the user’s likeness) of the respective user (e.g., replacing a two-dimensional image of the respective user or an avatar of the respective user with the user’s name, the user’s initials, and/or an image that is not based on the user’s likeness); and in accordance with a determination that the input directed to the affordance is detected while the second visual representation of the respective user is displayed, the computer system replaces display of the second visual representation of the respective user with the first visual representation of the respective user (e.g., replacing the user’s initials or the user’s name with an avatar of the respective user or a two-dimensional image of the respective user). This is described above, for example, in the description of
In some embodiments, while displaying the first user interface that includes the affordance for joining the first communication session, the computer system detects a third gaze input directed to the first user interface object (e.g., the third gaze input is detected after detecting movement of the user’s gaze away from the first user interface object after detecting the first gaze input). In some embodiments, in response to detecting the third gaze input directed to the first user interface object, the computer system displays the plurality of affordances for performing system operations associated with the first computer system without displaying the first user interface that includes the affordance for joining the first communication session. In some embodiments, after displaying the plurality of affordances for performing system operations associated with the first computer system without displaying the first user interface that includes the affordance for joining the first communication session, the first computer system detects that the user’s gaze is redirected to the first user interface object (e.g., after detecting movement of the user’s gaze away from the first user interface object after detecting the third gaze input). In response to detecting that the user’s gaze is redirected to the first user interface object (e.g., and in accordance with a determination that the request for the first computer system to join the first communication session satisfies the timing criteria), the first computer system ceases to display the plurality of affordances for performing system operations associated with the first computer system and redisplays the first user interface that includes the affordance for joining the first communication session.
In some embodiments, the computer system cycles through multiple user interfaces (e.g., a cycle that starts from the first user interface that includes the affordance for joining the first communication session, transitions to one or more intermediate user interfaces (e.g., a user interface that includes content associated with a first notification, as described herein with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator, sometimes appearing as a notification indicator) in the method 10000 has characteristics of the first user interface object (e.g., system control indicator) in the methods 8000-9000 and 11000-16000, and the user interface elements that are displayed (e.g., the additional content associated with a notification) may be replaced by, or concurrently displayed with, other user interface elements (e.g., the plurality of affordances for performing system operations associated with the first computer system, a user interface that includes an affordance for joining a communication session, and/or the user interface elements of the methods 8000-9000, and 11000-16000). For brevity, these details are not repeated here.
Initiating display of an environment-locked user interface from a viewpoint-locked user interface allows the computer system to display relevant user interfaces at appropriate locations in a view of a three-dimensional environment. This helps increase a level of immersion of a user of the computer system, as the user is not exposed to unnecessary affordances (e.g., for displaying or ceasing to display certain user interface elements, and/or for moving user interface elements) and does not need to perform additional user inputs in order to display user interface elements at appropriate positions in the view of the three-dimensional environment.
The computer system displays (11002), via the first display generation component (e.g., a display generation component of the computer system 7100, in
While displaying the first view of the three-dimensional environment, including displaying the first user interface element at the first position in the three-dimensional environment (e.g., the first position in the three-dimensional environment has the first spatial relationship with the first viewpoint of the user, and/or the first user interface element displayed with a first orientation and the first position has the first spatial relationship with the first viewpoint), the computer system detects (11004) first movement of a viewpoint of the user from the first viewpoint to a second viewpoint (e.g., movement of the user 7002 from the third location 7026-c to the fourth location 7026-d, in
In response to detecting the first movement of the viewpoint of the user from the first viewpoint to the second viewpoint (e.g., the spatial relationship between the viewpoint of the user and the three-dimensional environment changes as a result of the first movement), while a second view of the three-dimensional environment that corresponds to the second viewpoint of the user is visible via the one or more display generation components (e.g., the second view includes one or more virtual objects and, optionally, a representation of the physical environment that corresponds to the second viewpoint (e.g., a camera view, an image, and/or a view through a transparent or semitransparent portion of the first display generation component)), the computer system displays (11006) the first user interface element at a second position in the three-dimensional environment, wherein the first user interface element has the first spatial relationship with the second viewpoint of the user while the first user interface element is displayed at the second position in the three-dimensional environment (e.g., the first user interface element is viewpoint locked, and/or head locked; the spatial relationship between the first user interface element and the viewport through which the three-dimensional environment is visible does not change as a result of the movement of the viewpoint; and the spatial relationship between the first user interface element and the viewpoint does not change as a result of the first movement of the viewpoint) (e.g., the second position in the three-dimensional environment has the first spatial relationship with the second viewpoint of the user, and/or the first user interface element displayed with a second orientation and the second position has the first spatial relationship with the first viewpoint) (e.g., the system function menu 7024 is displayed with the same spatial relationship with the user’s viewpoint in both
While displaying the first user interface element in the second view of the three-dimensional environment (e.g., while displaying the first user interface element at the second position, with the first spatial relationship to the second viewpoint of the user and/or the viewport through which the three-dimensional environment is visible), the computer system detects (11008), via the one or more input devices, a first input that corresponds to activation of the first affordance (e.g., the first affordance is selected by a user input, such as an air tap or pinch in conjunction with a gaze input directed to the first affordance, a tap or pinch input at a location that corresponds to the position of the first affordance in the three-dimensional environment, or a gaze input directed to the first affordance detected in conjunction with a voice command to activate the first affordance) (e.g., the user’s attention 7116 is directed to the home affordance 7125 in
In response to detecting the first input that corresponds to activation of the first affordance, the computer system displays (11010) a second user interface element (e.g., an application window, a modal user interface, a search user interface, a control panel user interface, a communication user interface, and/or a home user interface) in the second view of the three-dimensional environment, wherein the second user interface element is displayed at a third position in the three-dimensional environment (e.g., the second user interface element is the user interface 7146, which is displayed in response to detecting the user’s attention 7116 directed to the home affordance 7124, in
While displaying the second view of the three-dimensional environment, including displaying the second user interface element at the third position in the three-dimensional environment (e.g., the third position in the three-dimensional environment has a second spatial relationship with the second viewpoint of the user, the third position in the three-dimensional environment has a second spatial relationship with the viewport through which the three-dimensional environment is visible, and/or the second user interface element displayed with a second orientation and the third position has the second spatial relationship with the second viewpoint) (e.g., in some embodiments, the first user interface element is displayed with a position and orientation that has the first spatial relationship with the second viewpoint and/or the viewport through which the three-dimensional environment is visible, or the first user interface element is no longer displayed in the second view of the three-dimensional environment), the computer system detects (11012) second movement of the viewpoint of the user from the second viewpoint to a third viewpoint (e.g., movement of the user 7002 from the fourth location 7026-d to the fifth location 7026-e, in
In response to detecting the second movement of the viewpoint of the user from the second viewpoint to the third viewpoint (e.g., the spatial relationship between the viewpoint of the user and the three-dimensional environment changes as a result of the second movement), while a third view of the three-dimensional environment that corresponds to the third viewpoint of the user is visible via the one or more display generation components (e.g., the third view includes one or more virtual objects and, optionally, a representation of the physical environment that corresponds to the third viewpoint (e.g., a camera view, an image, and/or a view through a transparent or semitransparent portion of the first display generation component)), the computer system displays (11014) the second user interface element at a location in the third view of the three-dimensional environment that corresponds to the third position in the three-dimensional environment (e.g., the second user interface element is world locked to the three-dimensional environment during the movement of the viewpoint of the user) (e.g., the user interface 7146 is displayed at the same position in the three-dimensional environment in
In some embodiments, the second user interface element is a search user interface that is configured to accept one or more search criteria from the user to perform a search. In some embodiments, while displaying the second user interface element, the computer system detects a first user input that includes one or more first search criteria. In some embodiments, in response to detecting the first user input that includes the one or more first search criteria, the computer system performs a first search in accordance with the one or more first search criteria (e.g., performing a search on the computer system, performing a system-level search beyond the scope of an individual application or experience, and/or performing a search on the Internet). In some embodiments, displaying the second user interface element includes displaying a search input field and, optionally, a soft keyboard in the search user interface displayed at the third position in the second view and the third view of the three-dimensional environment. In some embodiments, displaying the second user interface element includes displaying a voice input interface that accept one or more search criteria specified in the user’s speech input. In some embodiments, the search user interface accepts a drag and drop input that includes dragging and dropping a user interface element (e.g., text, document, image, webpage, a user’s avatar, an email address, a contact card, and/or a media item) from a portion of the three-dimensional environment that is outside of the search user interface (e.g., from a window and/or user interface of another application, and/or from a window of a communication session). In some embodiments, the search user interface includes application icons that corresponds to one or more applications and/or experiences, and/or avatars of one or more contacts of the user, that are automatically suggested by the computer system (e.g., based on the current context, based on the currently entered search criteria (e.g., full or partial search criteria), usage or search history).
For example, the space 7050 in
In some embodiments, detecting the first user input that includes the one or more first search criteria includes detecting a first gaze input that is directed to the second user interface element in conjunction with (e.g., before, and/or within a threshold amount of time of) detecting a first speech input that includes the one or more first search criteria, and performing a first search in accordance with the one or more first search criteria includes performing (optionally, in accordance with a determination that the first gaze input meets stability requirement (e.g., less than a threshold amount of movement within a threshold amount of time) within the second user interface element, or within a first portion of the second user interface element) the first search in accordance with the one or more first search criteria included in the first speech input. In some embodiments, the first gaze input is directed to a portion of the search user interface that corresponds to a search input field (e.g., a search bar). In some embodiments, the first gaze input is directed to a microphone icon displayed in the search user interface. In some embodiments, the computer system displays a visual change in the search user interface in accordance with a determination that the first gaze input has met the stability requirement, and optionally, displays a prompt for the user to speak to provide the search criteria. In some embodiments, the first speech input includes a natural language search query or a sequence of one or more search keywords rendered in human speech. In some embodiments, the computer system displays or otherwise outputs one or more search results that are responsive to the search query and/or search keywords. In some embodiments, the search results include application icons for one or more applications, and/or experiences, representations of one or more users, and/or one or more documents, images, and/or past communications. In some embodiments, the computer system displays visual feedback while the first speech input of the user is detected (e.g., a search bar of the search user interface updates to display text corresponding to the first speech input), and optionally updates in real time (e.g., the displayed text in the search bar of the search user interface updates to display a new word of the spoken search query, after a new word is spoken as part of the first speech input).
For example, in the descriptions of
In some embodiments, displaying the second user interface element that includes the search user interface includes a keyboard that is configured to enter one or more textual search keywords into the search user interface. In some embodiments, the keyboard is also displayed at the third position in the three-dimensional environment (e.g., or at another position that is immediately adjacent to the third position), and in response to detecting the second movement of the viewpoint of the user from the second viewpoint to the third viewpoint while a third view of the three-dimensional environment that corresponds to the third viewpoint of the user is visible via the one or more display generation components, the keyboard is displayed at the third position in the three-dimensional environment (e.g., or the other position that is immediately adjacent to the third position). In some embodiments, the keys in the keyboard are selectively activated by a gaze input on a respective key that is provided in conjunction with a tap or pinch gesture in the air. In some embodiments, the keys in the keyboard are selectively activated by a tap or pinch gesture at a location that corresponds to a respective key.
For example, in the description of
In some embodiments, the second user interface element includes representations of one or more notifications that have been received by the computer system. In some embodiments, the second user interface includes one or more notifications that are recently received and/or that have not been viewed by the user. In some embodiments, the second user interface includes a notification history interface that includes notifications that were previously displayed as new notifications and then stored in the notification history after a period of time or after a user has viewed and/or dismissed them. In some embodiments, the second user interface element includes multiple components, such as portions or all of a search user interface and portions or all of a notification user interface. In some embodiments, the second user interface element includes a single user interface, such as a search user interface or a notification user interface.
For example, the system space 7052 in
In some embodiments, the second user interface element includes at least a first icon corresponding to a first experience and a second icon corresponding to a second experience (e.g., the second user interface element is a home user interface that includes a plurality of application icons, which, when activated, cause the computer system to display respective applications corresponding to the plurality of application icons in the three-dimensional environment). In some embodiments, while displaying the second user interface element including the first icon and the second icon, the computer system detects a second user input directed to a respective icon of the first icon and the second icon (e.g., a gaze input directed to the first icon or the second icon, that is detected in conjunction with an air tap or air pinch gesture, or a tap or pinch gesture detected at a location that corresponds to the position of the first icon or the second icon in the three-dimensional environment). In some embodiments, in response to detecting the second user input directed to the respective icon and in accordance with a determination that the second user input is directed to the first icon, the computer system starts a first process to display the first experience in the three-dimensional environment. In some embodiments, in response to detecting the second user input directed to the respective icon and in accordance with a determination that the second user input is directed to the second icon, the computer system starts a second process to display the second experience in the three-dimensional environment. In some embodiments, the second user interface element is a home user interface includes a plurality of icons that correspond to a plurality of three-dimensional experiences in the three-dimensional environment (e.g., when activated, causes the system to display virtual sceneries, virtual worlds, and/or extended reality experiences). In some embodiments, the home user interface includes a plurality of icons that correspond to users, which, when activated, cause the computer system to display options for establishing communication sessions with the users in the three-dimensional environment. In some embodiments, the home user interface includes an initial user interface and one or more sub-level user interfaces that are displayed in response to interaction with user interface controls displayed in the initial user interface. In some embodiments, the primary user interface includes category icons for applications, experiences, and users, and optionally, a subset of frequently used application icons, experiences, and/or users; and the one or more sub-user interfaces display more application icons, icons for experiences, and representations of users, based on the selection of the category icons in the initial user interface. In some embodiments, the second user interface element includes multiple components, such as portions or all of a search user interface, portions or all of a notification user interface, and/or portions or all of a home user interface. In some embodiments, the second user interface element includes a single user interface, such as a search user interface, a notification user interface, or a home user interface.
For example, in the description of
In some embodiments, the first experience corresponds to a first application, and the second experience corresponds to a second application that is different from the first application. In some embodiments, displaying the first experience includes displaying an initial or default user interface of the first application in the three-dimensional environment; and displaying the second experience includes displaying an initial or default user interface of the second application in the three-dimensional environment. In some embodiments, the respective application user interface is displayed in a window of the respective application, and includes user interface elements that provide content and interactive functions of the respective application to the user. In some embodiments, the computer system ceases to display the second user interface element in response to detecting the second user input directed to the first icon or the second affordance and displaying the respective application user interface. In some embodiments, the respective application user interface is a world locked user interface in the three-dimensional environment.
For example, in the description of
In some embodiments, the first experience corresponds to a first other user (e.g., a first other user other than the user of the computer system), and the second experience corresponds to a second other user that is different than the first other user. In some embodiments, the second user interface element is the home user interface that includes respective representations of a plurality of users, which, when activated, cause the computer system to initiate respective communication sessions with the plurality of users. In some embodiments, the plurality of users are automatically selected by the system from a group of users that had communication with the user of the computer system in a recent period of time. In some embodiments, the plurality of users are automatically selected from a group of users that are stored in a contact list of the user of the computer system. In some embodiments, while displaying the second user interface element that includes the home user interface, the computer system detects user input that activates a first representation among the respective representations of the plurality of users (e.g., the third user input is a gaze input directed to the first representation detected in conjunction with an air tap or air pinch gesture, or a tap or pinch gesture detected at a location that corresponds to the position of the first representation in the three-dimensional environment); and in response to detecting the user input, the computer system initiates a first communication session with a first user that corresponds to the first representation. In some embodiments, initiating the first communication session includes displaying a plurality of selectable options that correspond to different communication protocols and/or applications (e.g., voice-only communication, video communication, extended reality communication, shared three-dimensional experience provided by a respective extended reality application, text messaging communication, email communication, and/or mixed modality communication involving two or more of the above), and establishing the communication session in accordance with the communication protocol and/or application that is selected by the user using the plurality of selectable options.
For example, in the description of
In some embodiments, the first experience corresponds to a first virtual three-dimensional environment, and the second experience corresponds to a second virtual three-dimensional environment that is different from the first virtual environment. In some embodiments, the second user interface element includes respective representations of a plurality of computer-generated three-dimensional experiences (e.g., virtual environments, virtual augmentation of the physical environments (e.g., virtual offices, virtual scenery (e.g., park, ocean, seaside, mountain, forest, beach, urban, dawn, and/or night), virtual mood or ambiance based on virtual lighting, décor, and/or sounds, virtual geographic locations, three-dimensional movies, and/or three-dimensional games), interactive experiences in an virtual and/or augmented reality environment). In some embodiments, while displaying the second user interface element including the home user interface, the computer system detects a user input that activates a second representation of the respective representation of the plurality of computer-generated three-dimensional experiences; and in response to detecting the user input, the computer system displays a first computer-generated three-dimensional experience that corresponds to the second representation. In some embodiments, displaying the first computer-generated three-dimensional experience includes replacing display of an existing virtual environment (e.g., a virtual environment that was displayed prior to detecting the user input) with a new virtual environment that corresponds to the first computer-generated three-dimensional experience. In some embodiments, displaying the first computer-generated three-dimensional experience includes replacing display of some or all of the virtual objects and content displayed in the three-dimensional environment with new virtual objects and content that correspond to the first computer-generated three-dimensional experience.
For example, in the description of
In some embodiments, displaying the second user interface element includes displaying a first portion of the second user interface element without displaying one or more second portions of the second user interface element (e.g., the second user interface element includes multiple sections or pages, and only one section or page is displayed at a time) (e.g., the multiple portions of the second user interface element correspond to a search user interface, a notification user interface, a control panel user interface, and/or a home user interface; or the multiple portions of the second user interface element correspond to a page for application icons, a page for experiences, and a page for users). In some embodiments, the computer system optionally displays one category of objects (e.g., categories such as search-related objects, notification-related objects, system-control related objects, and/or home-related objects) in the second user interface element at a time. In some embodiments, the computer system optionally displays one main category of objects (e.g., in the central region of the second user interface element), and reveals a hint of other category of objects at the same time (e.g., on the sides or other peripheral region of the second user interface element, in a smaller size, or with reduced luminance, color, and/or opacity, as compared to the objects in the central portion of the second user interface element). In some embodiments, the computer system optionally displays a subset, less than all, of all icons of a respective category of icon at a given time. In some embodiments, while displaying the first portion of the second user interface element without displaying the one or more second portions of the second user interface element, the computer system detects a third user input. In some embodiments, in response to detecting the third user input and in accordance with a determination that the third user input meets first criteria (e.g., the third user input meets switching criteria or meets scrolling criteria), the computer system ceases to display the first portion of the second user interface element and displaying at least one of the one or more second portions of the second user interface element. In some embodiments, different portions of the second user interface element include different types of icons. For example, the first portion of the second user interface element includes application icons that activate experiences corresponding to different applications, a second portion of the second user interface element includes user icons that activate experiences corresponding to different other users, and/or a third portion of the second user interface element includes environment icons that activate experiences corresponding to different virtual environment. Performing the third user input allows the user to switch between displaying application icons and displaying user icons (and/or environment icons). In some embodiments, the different portions of the second user interface element include different subsets of the same type of icons, and scrolling or switching between the different portions causes display of the different subsets of a respective type of icons. In some embodiments, the one or more second portions of the second user interface element include a third portion of the second user interface element and a fourth portion of the second user interface element, and the computer system displays the third portion of the second user interface element in response to detecting the third user input that meets the first criteria, and in response to detecting a fourth user input that meets the first criteria (e.g., a fourth user input that is the same as the third user input), the computer system ceases to display the third portion of the second user interface element and displays the fourth portion of the second user interface element. In some embodiments. In some embodiments, the user can continue to perform user inputs that meet the first criteria to cycle between the different portions of the second user interface element (e.g., from the first portion to the third portion, from the third portion to the fourth portion, from the fourth portion to the first portion, and so on). In some embodiments, the user can perform a user input that meets second criteria different from the first criteria, to reverse the order through which the computer system displays the portions of the second user interface element. In some embodiments, the first criteria require that the third user input includes a swipe gesture in a first direction in order for the first criteria to be met, and the second criteria require that the (subsequent) user input includes a swipe gesture in a second direction that is opposite the first direction, in order for the second criteria to be met. In some embodiments, displaying the at least one of the one or more second portions of the second user interface element includes replacing display of the first portion of the second user interface element with said at least one of the second portions of the second user interface element in the second user interface element (e.g., in the central portion of the second user interface element). In some embodiments, the third user input that meets the first criteria is a navigation gesture in a first direction or a second direction, and the computer system navigates (e.g., switches or scrolls) to the next portion of the second user interface element in that first direction or second direction. As used herein, a “navigation gesture” can be any suitable navigation gesture (e.g., for navigating between different portions/sections of a user interface, or for navigating between different user interfaces). In some embodiments, the navigation gesture is a pinch and drag air gesture. In some embodiments, the navigation gesture is a gesture that activates an affordance (e.g., a left or right arrow affordance, or an affordance that includes text (e.g., “next,” “next section,” or analogous text)) for navigating between portions of the second user interface element. In some embodiments, the swipe gesture is a gaze gesture directed to a specific region (e.g., a left or right edge/boundary) of the second user interface element. In some embodiments, the computer system shifts the content displayed in the second user interface element to switch out the currently displayed portion of the second user interface element, and shift in the next portion of the second user interface element in the first direction or second direction. In some embodiments, the third user input that meets the switching criteria includes an air tap input or air pinch input directed to a respective category icon corresponding to a respective category of icons in the home user interface or a paging indicator corresponding to a page or section of the home user interface. In some embodiments, the third user input that meets the switching criteria includes a swipe input (e.g., air swipe, or swipe on a touch-sensitive surface) in the left and right direction (or alternatively, in the up and down direction) that causes the computer system to switch between displaying different categories of icons (e.g., icons for applications, experiences, and users) in the home user interface. In some embodiments, the third user input that meets the scrolling criteria includes an air tap or air pinch input detected in conjunction with a gaze input directed to a scrolling bar associated with the currently displayed category of icons and/or page of the second user interface element. In some embodiments, the third user input that meets the scrolling criteria includes a swipe input (e.g., air swipe, or swipe on a touch-sensitive surface) in the up and down direction (or alternatively, in the left and right direction) that causes the computer system scroll through the currently displayed category of icons or the currently displayed page or section of the second user interface element.
For example, in the description of
In some embodiments, the second user interface element includes a home user interface. In accordance with the determination that the third user input meets the first criteria, the computer system ceases to display the first portion of the second user interface element and displays at least one of the one or more second portions of the second user interface element. In accordance with a determination that the third user input meets switching criteria (e.g., the third user input selects a respective category icon for a category of icons, and/or selects a page or section indicator displayed in the second user interface element; or the third user input is a swipe gesture in a first direction (e.g., up and down direction, or alternatively, left and right direction)), the computer system ceases to display a first section of the home user interface and displays a second section of the home user interface, wherein the first section of the home user interface and the second section of the home user interface correspond to different categories of icons selected from a first category corresponding to applications, a second category corresponding to computer-generated three-dimensional experiences, and a third category corresponding to users. In some embodiments, in response to detecting a first type of air gesture (e.g., a swipe gesture in a first direction (e.g., horizontal direction, left and right direction, or up and down direction)), or a combination of a gaze input on the home user interface and the first type of air gesture), the computer system switches between displaying different sections of the home user interface (e.g., applications, experiences, and users).
For example, in the description of
In some embodiments, in accordance with the determination that the third user input meets the first criteria, the computer system ceases to display the first portion of the second user interface element and displaying at least one of the one or more second portions of the second user interface element. In accordance with a determination that the third user input meets scrolling criteria (e.g., the third user input activates a scroll control in the home user interface; or the third user input is a swipe gesture in a second direction (e.g., left and right direction, or alternatively, up and down direction)) different from the switching criteria, the computer system scrolls through a currently displayed section of the home user interface (e.g., ceases to display a first portion of the currently displayed section of the home user interface and displays a second portion of the currently displayed section of the home user interface, wherein the first portion and the second portion of the currently displayed section of the home user interface include different subsets of the category of icons corresponding to the currently displayed section of the home user interface (e.g., different subsets of icons from the first category corresponding to applications, different subsets of icons from the second category corresponding to computer-generated three-dimensional experiences, or different subsets of icons from the third category corresponding to users). In some embodiments, in response to detecting a second type of air gesture (e.g., a swipe gesture in a second direction (e.g., vertical direction, up and down direction, or left and right direction)), or a combination of a gaze input on a scroll bar in the home user interface and the first type of air gesture), the computer system scrolls through different subsets of icons in the currently displayed section of the home user interface (e.g., applications, experiences, and users).
For example, in the description of
In some embodiments, the first portion of the second user interface element includes a first set of icons (e.g., application icons, icons for starting different experiences, icons corresponding to different users) and the at least one of the one or more second portion of the second user interface element includes a second set of icons (e.g., application icons, icons for starting different experiences, icons corresponding to different users) different from the first set of icons. The respective icons of the first set of icons are displayed with different values for a first visual property (e.g., size, color, luminance, opacity, thickness, degree of fading, degree of blurring, and/or visual depth) while the first portion of the second user interface element is displayed. The respective icons of the second set of icons are displayed with different values for a second visual property (e.g., size, color, luminance, opacity, thickness, degree of fading, degree of blurring, and/or visual depth) (e.g., same as the first visual property, or different from the first visual property) while the at least one of the one or more second portions of the second user interface element is displayed. For example, in some embodiments, when the application section of the home user interface is displayed, application icons that are displayed in the central region of the home user interface are displayed with larger sizes, greater luminance, greater opacity, greater three-dimensional thickness, less fading, less blurring, and/or smaller visual depth than application icons that are displayed closer to the edge of the home user interface. In some embodiments, when switching between different sections of the home user interface, as some icons move toward the central section of the home user interface, they become larger, brighter, more opaque, thicker, less faded, less blurred, and/or move closer to the viewpoint; and as other icons move away from the central section of the home user interface toward the edge of the home user interface, they become smaller, dimmer, less opaque, thinner, more faded, more blurred, and/or move farther away from the viewpoint.
For example, in the description of
In some embodiments, the first set of icons includes at least a first icon in a central portion of the first portion of the second user interface element, and a second icon in a peripheral portion of the first portion of the second user interface element. In some embodiments, the first icon is displayed with a first value of the first visual property, the second icon is displayed with a second value of the first visual property, and the first value of the first visual property corresponds to more visual emphasis than the second value of the first visual property (e.g., the first value corresponds to a larger size, greater luminance, greater opacity, greater three-dimensional thickness, less fading, less blurring, and/or smaller visual depth than the second value of the first visual property). In some embodiments, the second set of icons includes at least a third icon in a central portion of the at least one of the second portions of the second user interface element, and a fourth icon in a peripheral portion of the at least one of the second portions of the second user interface element. In some embodiments, the third icon is displayed with a third value of the first visual property, the fourth icon is displayed with the fourth value of the first visual property, and the third value of the first visual property corresponds to more visual emphasis than the fourth value of the first visual property (e.g., the first value corresponds to a larger size, greater luminance, greater opacity, greater three-dimensional thickness, less fading, less blurring, and/or smaller visual depth than the second value of the first visual property).
For example, in the description of
In some embodiments, the first icon (e.g., as a representative icon of icons in the central portion of the first portion of the second user interface element) is displayed with a greater thickness in a direction of visual depth in the three-dimensional environment than the second icon (e.g., as a representative icon of icons in the peripheral or edge portions of the first portion of the second user interface element). In some embodiments, the first icon (e.g., as a representative of icons in the central portion of the first portion of the second user interface element) is displayed at a smaller visual depth from the viewpoint of the user than the second icon (e.g., as a representative of icons in the peripheral or edge portions of the first portion of the second user interface element). In some embodiments, the first icon (e.g., as a representative of icons in the central portion of the first portion of the second user interface element) is displayed at a greater visual depth from the viewpoint of the user than the second icon (e.g., as a representative of icons in the peripheral or edge portions of the first portion of the second user interface element). In some embodiments, the icons in the first portion of the second user interface element are distributed on an invisible concave or convex surface (e.g., curving toward the viewpoint, or curving away from the viewpoint) in the three-dimensional environment, and the icons that are in the central portion of the field of view are displayed with greater thickness in the direction of visual depth than the icons that are in the peripheral or edge portions of the field of view (e.g., icons near the edge of the second user interface element or the field of view are flattened more and increasingly two-dimensional as their positions become farther away from the central region of the second user interface element or the field of view).
For example, in the description of
In some embodiments, before displaying the first user interface element, including at least the first affordance, at the first position in the three-dimensional environment, the computer system displays a third user interface element at the fourth position (e.g., at or near the first position) in the three-dimensional environment (e.g., the third user interface element is a small icon or indicator (e.g., of system function menu) that moves with the viewpoint, e.g., remains in the top portion of the field of view, or is displayed in response to the user’s gaze in the top portion of the field of view), wherein the third user interface element does not include the first affordance (e.g., the third user interface element does not include any affordance, or the third user interface element does not include the affordances for triggering display of the system user interfaces, such as the home user interface, notification user interface, search user interface, multitasking user interface, and/or control panel user interface). In some embodiments, while displaying the third user interface element at the fourth position in the three-dimensional environment, the computer system detects a fourth user input that meets second criteria (e.g., the fourth user input includes a tap or pinch gesture directed to the third user interface element, optionally, while a gaze input is directed to the third user interface element; or the fourth user input is a gaze input directed to the third user interface element for at least a threshold amount of time). In some embodiments, in response to detecting the fourth user input that meets the second criteria, the computer system displays the first user interface element at the first position in the three-dimensional environment. In some embodiments, displaying the first user interface element at the first position in the three-dimensional environment, in response to detecting the fourth user input, includes displaying an animation of the first user interface element expanding outward from the third user interface element. In some embodiments, displaying the first user interface element at the first position in the three-dimensional environment, in response to detecting the fourth user input, includes replacing display of the third user interface element with display of the first user interface element. In some embodiments, the first user interface element and the third user interface element are concurrently displayed.
For example, in
In some embodiments, the fourth user input that meets the second criteria includes a gaze input directed to the third user interface element (e.g., a gaze input that remains within a threshold distance of the third user interface element for more than a threshold amount of time). For example, in
In some embodiments, the fourth user input includes a hand gesture (e.g., an air gesture (e.g., an air tap or an air pinch), a tap gesture, a pinch gesture, and/or other gestures) performed in conjunction with (e.g., within a threshold amount of time before, within a threshold amount of time after, or concurrently with) a gaze input directed to the third user interface element. For example, in
In some embodiments, the first user interface element includes a second affordance different from the first affordance. In some embodiments, while the second user interface element is at the third position in the three-dimensional environment (e.g., visible within the currently displayed view, or outside of the currently displayed view), the computer system displays the first user interface element at a fifth position in the three-dimensional environment (e.g., in response to a gaze input directed to the third user interface element, or a combination of a gaze input directed to the third user interface element and a pinch or tap gesture), wherein the first user interface element displayed at the fifth position (and with a respective orientation) has the first spatial relationship to the third viewpoint of the user (e.g., the first user interface element is viewpoint locked and moves with the viewpoint). In some embodiments, while the second user interface element is at the third position in the three-dimensional environment and while the first user interface element is displayed at the fifth position in the three-dimensional environment, the computer system detects a seventh user input that corresponds to activation of the second affordance. In some embodiments, in response to detecting the seventh user input that corresponds to activation of the second affordance, the computer system displays a fourth user interface element (e.g., a search user interface, a home user interface, a control panel user interface, a notification user interface, and/or a multitasking user interface, any of the user interfaces described herein that is different from the second user interface element) at a sixth position in the three-dimensional environment, wherein the fourth user interface element is different from the second user interface element. For example, in some embodiments. different user interfaces (e.g., system level user interfaces, application-level user interfaces, experiences, and/or communication user interfaces) are optionally displayed as world locked user interfaces in response to activation of respective affordances displayed in the first user interface element.
For example, in
In some embodiments, the seventh user input is detected while the first user interface element and the second user interface element are concurrently visible in a currently displayed view of the three-dimensional environment (e.g., the third view of the three-dimensional environment, the second view of the three-dimensional environment, or another view of the three-dimensional environment that corresponds to the current viewpoint of the user). For example, in
In some embodiments, the computer system ceases to display the second user interface element in response to detecting the seventh user input (e.g., replaces display of the second user interface element at the third position in the three-dimensional environment with display of the fourth user interface element at the third position in the three-dimensional environment; or ceases display of the second user interface element at the third position in the three-dimensional environment, and displays the fourth user interface element at the sixth position in the three-dimensional environment). In some embodiments, the fourth user interface element has properties (e.g., movement and/or appearance properties) and user interactions (e.g., activation, navigation, switching, and/or scrolling) analogous to those described herein with respect to the second user interface element, which are not repeated in the interest of brevity.
For example, in the description of
In some embodiments, while displaying the first view of the three-dimensional environment, including displaying the first user interface element at the first position and displaying the second user interface element at the second position in the three-dimensional environment, the computer system detects third movement of the viewpoint of the user from the first viewpoint to a fourth viewpoint. In some embodiments, in response to detecting the third movement of the viewpoint of the user from the first viewpoint to the fourth viewpoint, the computer system displays a fourth view of the three-dimensional environment that corresponds to the fourth viewpoint of the user, and the computer system displays the first user interface element at a seventh position in the three-dimensional environment, wherein the first user interface element has the first spatial relationship with the fourth viewpoint of the user (e.g., the first user interface element is viewpoint locked, and/or head locked) while the first user interface element is displayed at the seventh position in the three-dimensional environment. The computer system displays the second user interface element at the second position in the three-dimensional environment (e.g., the second user interface element is world locked to the three-dimensional environment).
For example, in
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 11000 in some circumstances has a different appearance as described in the methods 9000-10000, and 12000-16000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-10000, and 12000-16000). For brevity, these details are not repeated here.
Displaying (e.g., based on the attention criteria) a first user interface object, in response to detecting a first user input that includes a first gaze input directed to a first position, and in accordance with a determination that the first position in the three-dimensional environment has a first spatial relationship to a viewport through which the three-dimensional environment is visible, and forgoing displaying the first user interface object in the first view of the three-dimensional environment, in accordance with a determination that the first position in the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment is visible, reduces the number of inputs needed to access system functions of the computer system without cluttering the user interface with additional displayed controls, user interfaces, and/or user interface objects.
While a first view of a three-dimensional environment is visible via the first display generation component, the computer system detects (12002), via the one or more input devices, a first user input (e.g., a gaze input represented by the user’s attention 7116 in
In response to detecting (12004) the first user input including detecting the first gaze input, and in accordance with a determination that the first position in the three-dimensional environment has a first spatial relationship to a viewport (e.g., the display of the computer system 7100 in
In response to detecting the first user input including detecting the first gaze input, and in accordance with a determination that the first position in the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment his visible (e.g., a determination that the first position is not represented in the first region of the field of view provided by the first display generation component (e.g., in the upper left corner, top center region, lower right corner, peripheral region, or other preselected region of the field of view), while the first view of the three-dimensional environment is visible in the field of view), the computer system forgoes (12008) displaying the first user interface object in the first view of the three-dimensional environment (e.g., in
In some embodiments, while the first user interface object is not visible in a currently displayed view (e.g., the first view or another view displayed after the first view) of the three-dimensional environment (e.g., after the first user interface object is dismissed from the first view of the three-dimensional environment), the computer system detects a first change of a viewpoint of a user from a first viewpoint associated with the first view of the three-dimensional environment to a second viewpoint associated with a second view of the three-dimensional environment (e.g., based on movement of at least a portion of the computer system and/or a movement of a portion of the user which is the basis for determining the viewpoint of the user). In response to detecting the first change in the viewpoint of the user, the computer system updates the currently displayed view of the three-dimensional environment in accordance with the first change in the viewpoint of the user, to display the second view of the three-dimensional environment; In some embodiments, the change in the current viewpoint of the user (e.g., from the first viewpoint to the second viewpoint) is accomplished by moving the first display generation component and/or the one or more cameras in the physical environment, and/or movement of the user (e.g., turning, walking, running, and/or tilting the head up or down) in the physical environment that change the pose (e.g., position and/or facing direction) of the user relative to the three-dimensional environment. While the second view of the three-dimensional environment is visible via the first display generation component, the computer system detects, via the one or more input devices, a second user input, including detecting a second gaze input that is directed to a third position, different from the first position, in the three-dimensional environment. In response to detecting the second user input including detecting the second gaze input and in accordance with a determination that the third position in the three-dimensional environment has the first spatial relationship to the viewport through which the three-dimensional environment is visible (e.g., a determination that the third position is represented in the first region of the field of view provided by the first display generation component (e.g., in the upper left corner, top center region, lower right corner, peripheral region, or other preselected region of the field of view), while the second view of the three-dimensional environment is visible in the viewport), the computer system displays the first user interface object in the second view of the three-dimensional environment, at a fourth position in the three-dimensional environment that has the second spatial relationship to the second view of the three-dimensional environment; and In some embodiments, the fourth position is offset from the third position in the second view in the same manner as how the second position is offset from the first position in the first view. In response to detecting the second user input including detecting the second gaze input and in accordance with a determination that the third position in the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment is visible (e.g., a determination that the second position is not represented in the first region of the field of view provided by the first display generation component (e.g., in the upper left corner, top center region, lower right corner, peripheral region, or other preselected region of the field of view), while the second view of the three-dimensional environment is visible in the viewport), the computer system forgoes displaying the first user interface object in the second view of the three-dimensional environment. In some embodiments, the first user interface object is not displayed in the second view of the three-dimensional environment until a gaze input that is directed to a position that has the first spatial relationship to the second view of the three-dimensional environment has been detected.
For example, as described with reference to
In some embodiments, in response to detecting the first user input including detecting the first gaze input: in accordance with a determination that the first position in the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment is visible (e.g., a determination that the first position is not represented in the first region of the field of view provided by the first display generation component (e.g., in the upper left corner, top center region, lower right corner, peripheral region, or other preselected region of the field of view), while the first view of the three-dimensional environment is visible in the viewport) and that a second user interface object, different from the first user interface object, occupies the first position in the three-dimensional environment, the computer system performs a respective operation that corresponds to the second user interface object. In some embodiments, the second user interface object is an application user interface of a respective application or an affordance within the application user interface, and the first computer system performs an operation within the respective application (e.g., including updating the content included in the application user interface, displaying additional user interface objects in the three-dimensional environment, ceasing to display the first user interface object, and other operations of the respective application)). In some embodiments, the second user interface object is an object that accepts the first user input as a valid or reacts to the first user input, and the first computer system performs the operation in accordance with the instructions associated with the second user interface object for the input type and characteristics of the first user input.
For example, as described with reference to
In some embodiments, while the first view of the three-dimensional environment is visible and the first user interface object is not displayed in the first view of the three-dimensional environment, the computer system detects a third user input that includes a third gaze input that is directed to a fifth position in the three-dimensional environment (e.g., the third user input precedes the first user input, and optionally, the first user input is a continuation of the third user input (e.g., the gaze of the user moves from the fifth position to the first position)). In response to detecting the third user input that includes the third gaze input: in accordance with a determination that the fifth position in the three-dimensional environment is within a first region that includes a respective position having the first spatial relationship to the viewpoint through which the three-dimensional environment is visible (e.g., the first region includes the respective position for triggering display of the first user interface object and a respective area or volume surrounding the respective position (e.g., a rectangular area, a spherical volume, or other shapes, with respective boundaries relative to the respective position)), the computer system displays a third user interface object (e.g., a visual indicator (e.g., the indicator 7010 of system function menu of
For example, in
In some embodiments, the first region includes a first subregion including the respective position that has the first spatial relationship to the viewpoint through which the three-dimensional environment is visible and a second subregion (e.g., a region surrounding the first subregion, or adjacent to the first subregion) that does not include the respective position. For example, in some embodiments, the first region is a rectangular region in the field of view that encloses a smaller circular region, where the gaze input directed to the rectangular region outside of the smaller circular region causes the first computer system to, optionally, display a visual indicator inside the smaller circular region, and a gaze input directed to the visual indicator or the smaller circular region (e.g., with, or without the visual indicator inside) causes the first computer system to display the first user interface object that includes the affordances for the set of functions. For example, in
In some embodiments, displaying the first user interface object at the second position in response to detecting the first user input including the first gaze input is further in accordance with a determination that the first gaze input is maintained within the first subregion for at least a first threshold amount of time (e.g., at the first position, on the second user interface object, and/or within a threshold distance of the first position). In some embodiments, if the first gaze input moves outside of the first subregion before the first threshold amount of time is reached, the first computer system does not display the first user interface object at the second position in the three-dimensional environment. In some embodiments, if the first gaze input moves outside of the first subregion before the first threshold amount of time is reached but the first computer system detects another confirmation gesture while the first gaze input is maintained on the first subregion, the first computer system still displays the first user interface object at the second position in the three-dimensional environment. For example, as described with reference to
In some embodiments, while the first user interface object is not visible in the first view of the three-dimensional environment (e.g., after the first user interface object is dismissed from the first view of the three-dimensional environment), the computer system detects, via the one or more input devices, a fourth user input, including detecting a fourth gaze input that is directed to the first subregion and that has not been maintained within the first subregion for at least the first threshold amount of time. In response to detecting the fourth user input including the fourth gaze input, and in accordance with a determination that a respective gesture meeting first criteria has been detected while the fourth gaze input is maintained in the first subregion, the computer system displays the first user interface object at the second position in the three-dimensional environment (e.g., if a pinch gesture is detected while the user gazes at the smaller region or the indicator (e.g., of system function menu), the first computer system displays the first user interface object before the first threshold amount of time is reached). In some embodiments, in response to detecting the fourth user input, in accordance with a determination that a respective gesture meeting the first criteria has not been detected while the fourth gaze input is maintained in the first subregion, the first computer system forgoes displaying the first user interface object at the second position in the three-dimensional environment before the third gaze input has been maintained in the first subregion for at least the first threshold amount of time. In some embodiments, the first position of the three-dimensional environment that has the first spatial relationship to the viewport through which the three-dimensional environment is visible while the viewport through which the three-dimensional environment is visible, is a representative of a respective position that has the first spatial relationship to the viewport through which the three-dimensional environment is visible; and the second position that has the second spatial relationship to the first view of the three-dimensional environment while the first view of the three-dimensional environment is visible, is a representative of a respective position that has the second spatial relationship to the viewport through which the three-dimensional environment is visible. The respective positions represented by the first position and the second position are updated as the currently displayed view is updated based on the changes in the viewpoint of the user. For example, as described with reference to
In some embodiments, the first user interface object includes a respective system user interface (e.g., the system function menu 7024 of
In some embodiments, while displaying the first user interface object, in the first view of the three-dimensional environment, at the second position in the three-dimensional environment that has the second spatial relationship to the first view of the three-dimensional environment, the computer system detects that user attention is no longer directed to the first user interface object (e.g., the first gaze input is no longer directed to and/or has moved away from the first position, the first subregion, and/or the first user interface object, in the three-dimensional environment). In response to detecting that the user attention is no longer directed to the first user interface object (e.g., for at least a threshold duration of time, such as 1 second, 5 seconds, 10 seconds, or 30 seconds), the computer system ceases to display the first user interface object in the first view of the three-dimensional environment. In some embodiments, the first view is a representative of a currently displayed view, and the behavior of the first computer system is analogous to what is described with respect to the first view, when another view is the currently displayed view of the three-dimensional environment. For example, this is described with reference to
In some embodiments, the determination that the first position in the three-dimensional environment has the first spatial relationship to the viewport through which the three-dimensional is visible environment includes a determination that the first position is within a first response region of a first size, and detecting that the attention of the user is no longer directed to the first position in the three-dimensional environment includes detecting that the user attention has moved from within the first response region (e.g., to within the second response region and then) to outside of a second response region of a second size that is different from (e.g., larger, or smaller) the first size. In some embodiments, the second response region encloses the first response region entirely or partially. In some embodiments, the first response region corresponds to the first region used to trigger the display of the first user interface object (e.g., the system function menu 7024 of
In some embodiments, while displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects a fourth user input including detecting gaze input directed to a respective affordance of the one or more affordances for accessing the first set of functions of the first computer system in conjunction with detecting a first speech input from a user. In response to detecting the fourth user input, the computer system performs a respective operation corresponding to the respective affordance in accordance with the first speech input. In some embodiments, in accordance with a determination that the respective affordance is a first affordance, the first computer system performs a first operation corresponding to the first affordance in accordance with the first speech input; and in accordance with a determination that the respective affordance is a second affordance, the first computer system performs a second operation corresponding to the second affordance in accordance with the first speech input. In some embodiments, in response to detecting the first speech input, the computer system displays a visual indication corresponding to the detected speech input (e.g., the computer system displays one or more words corresponding to, or detected in, the speech input). In some embodiments, the computer system re-performs (or continues performing) the function of the first computer system corresponding to the respective affordance if (or as) the speech input continues. For example, if the function of the first computer system is a search function, the computer system performs the search function once a first word (e.g., “Apple”) of the speech input is detected (e.g., displays search results corresponding to the first word “Apple” of the speech input). If the computer system detects that the speech input continues (e.g., the user continues to speak/provide the verbal user input), the computer system updates the displayed search results to display results corresponding to the new/updated speech input (e.g., “Apple Park”). In some embodiments, the visual appearance of the respective appearance changes in accordance with various characteristics of the first speech input, e.g., changes in brightness, color, and other visual properties in accordance with the pitch, volume, and/or other characteristics of the speech input, as the speech input is received. For example, as described with reference to
In some embodiments, performing the respective operation corresponding to the respective affordance in accordance with the first speech input includes, in accordance with a determination that the respective affordance is an affordance for accessing a virtual assistant function of the first computer system, performing an operation corresponding to instructions contained in the first speech input (e.g., as interpreted by a virtual assistant corresponding to the virtual assistant function of the first computer system).In some embodiments, the first speech input includes instructions for performing a search (e.g., includes a spoken search term for an Internet search, or a search of documents and/or media stored in memory of the computer system), and the virtual assistant performs the search and presents a plurality of search results corresponding to the search. In some embodiments, the first speech input includes instructions for launching one or more applications (e.g., a music application, a notetaking application, or an alarm/timer/clock application), and the virtual assistant launches the one or more applications, and optionally performs an additional functions associated with the one or more applications (e.g., plays a song corresponding to a spoken song name included in the first speech input, creates a note including spoken information included in the first speech input, sets an alarm or timer based on a spoken time/time period included in the first speech input). In some embodiments, the first speech input includes instructions for initiating a communication session (e.g., a phone call, a video call, or a shared XR experience) with one or more other users, and the virtual assistant initiates the communication session (e.g., with the one or more other users, based on one or more spoken names and/or contact identifiers, included in the first speech input). In some embodiments, the first speech input includes instructions for sending a text or email message, and the virtual assistant sends the text or email message (e.g., to one or more other users, based on one or more spoken names and/or contact identifiers, included in the first speech input) (e.g., including a spoken message included in the first speech input). In some embodiments, the first speech input includes instructions for adjusting one or more settings of the computer system, and the virtual assistant adjusts the one or more settings (e.g., based on a spoken setting and/or desired adjustment (e.g., increasing a brightness of the display, lowering a volume of the computer system, and/or setting a do-not-disturb setting of the computer system)). For example, as described with reference to
In some embodiments, performing the respective operation corresponding to the respective affordance in accordance with the first speech input includes, in accordance with a determination that the respective affordance is an affordance for accessing a text entry function (e.g., a search function, messaging function, or other functions accepting textual inputs) of the first computer system that accepts text input (e.g., a search term that is searched when the computer system performs the search function), providing text converted from the first speech input as input to the text entry function (e.g., and optionally displays visual feedback regarding the speech user input (e.g., displays text corresponding to the one or more words detected in the verbal user input in a search field, or other field configured to receive text)). For example, as described with reference to
In some embodiments, while displaying the first view of the three-dimensional environment via the first display generation component, the computer system determines a current spatial relationship between the first display generation component and a user (e.g., which affects the current spatial relationship between the field of view and the eyes of the user). In some embodiments, the display generation component is a head-mounted display that is worn with various positions and/or with orientations (e.g., angle, relative to the user’s eyes or face) on the user’s head. In some embodiments, the display generation component is a handheld device, a watch, or other electronic device worn on the wrist, arm, or hand that has various spatial relationships to the user or a particular portion of the user (e.g., the user’s eyes, hand, wrist, or other relevant portions). In some embodiments, the first display generation component is a heads-up display that is mounted at different angles and/or orientations relative to the user. The computer system adjusts criteria for determining whether the respective position has the first spatial relationship to the viewport through which the three-dimensional environment is visible in accordance with the current spatial relationship between the first display generation component and the user (e.g., using different criteria for different spatial relationships). In some embodiments, in a first scenario, the first display generation component has a third spatial relationship to the user, and in a second scenario, the first display generation component has a fourth spatial relationship to the user, wherein the fourth spatial relationship differs from the third spatial relationship by an offset amount (e.g., in terms of distance, angles, and/or orientation); the respective position that meets the criteria for having the first spatial relationship to the first view would appear in different portions of the field of view provided by the first display generation component, such that the user does not have to strain uncomfortably to look at the same position to invoke the first user interface object irrespective of how the first display generation component is placed relative to the user. In a more specific example where the first display generation component is a head-mounted display that can be worn and/or positioned different for different users (e.g., in order to achieve a light seal around the user’s eyes and/or face, and/or due to different physical features or geometry of different users’ faces). If the head-mounted display sits further out on the user’s nose, the display of the head-mounted display is lower, relative to the user’s eyes (e.g., as compared to if the head-mounted display sat at a position on the user’s nose that was closer to the user’s face). If a default first spatial relationship required the first position to be a first distance from a top edge of the display, the computer system accommodates for the user’s particular way of wearing the head-mounted display by adjusting the criteria for determining whether the respective position has the first spatial relationship to the field of view, and requiring the first position to be a second distance (e.g., that is less than the first distance) from the top edge of the display, in accordance with some embodiments. Stated differently, if the first spatial relationship required the user’s attention to be directed to a first reactive region in the displayed field of view of the three-dimensional environment by default, the computer system accommodates for the user’s particular way of wearing the head-mounted display by moving the first reactive region upwards (e.g., relative to the display of the display generation component itself). In some embodiments, the first reactive region is adjusted upwards, downwards, to the left, and/or to the right, depending on position of the computer system (e.g., how a head-mounted display is worn) relative to the body or eyes of the user who is in a position to view the three-dimensional environment via the display generation component. In some embodiments, the amount of adjustment is comparatively small (e.g., to accommodate for natural asymmetry of the user’s face). In some embodiments, the amount of adjustment is comparatively large (e.g., to accommodate intentional user choices regarding how the computer system is worn and/or positioned). For example, as described with reference to
In some embodiments, in accordance with a determination that the current spatial relationship between the first display generation component and the user no longer meets alignment criteria (e.g., based on the determined current spatial relationship between the first display generation component and the user) (e.g., the computer system is located beyond a threshold distance from the user, the computer system is positioned with too large an angle relative to the user, a head-mounted display is worn too high, too low, or too far to the right or left, relative to the user’s eyes or face, and/or a head-mounted display tilts too far forwards or backwards (e.g., because the head-mounted display is worn too loosely on the user’s head)), the computer system displays a second visual indication that the current spatial relationship between the first display generation component and the user no longer meets the alignment criteria. In some embodiments, the second visual indication further instructs the user to change the current spatial relationship between the first display generation component and the user. In some embodiments, the first computer generated component forgoes displaying the first user interface object until the alignment criteria are met after adjustment to the current spatial relationship between the first display generation component and the user. For example, as described with reference to
In some embodiments, displaying the first user interface object at the second position that has the second spatial relationship to the viewport through which the three-dimensional environment is visible includes adjusting criteria for establishing the second spatial relationship between the first user interface object and the viewport through which the three-dimensional environment is visible in accordance with the current spatial relationship between the first display generation component and the user (e.g., using different criteria for different spatial relationships between the first display generation component and the user).For example, as described with reference to
In some embodiments, the computer system displays one or more user interface objects in the first view of the three-dimensional environment, wherein the one or more user interface objects (e.g., the second user interface object, or other objects located at other positions in the three-dimensional environment) (e.g., application user interfaces, other system user interface objects, virtual background of the three-dimensional environment, and/or representations of physical objects in the physical environment) are different from the first user interface object (and different from the third user interface object (e.g., the indicator 7010 of system function menu of
In some embodiments, at a first time, the one or more affordances for accessing the first set of functions of the first computer system include a first affordance for adjusting an audio level of the first computer system, and at a second time, different from the first time, the one or more affordances for accessing the first set of functions of the first computer system include a second affordance for adjusting an audio level of a first type of audio provided by the first computer system and a third affordance for adjusting an audio level of a second type of audio provided by the first computer system, wherein the second affordance and the third affordance are different from the first affordance. In some embodiments, the second affordance and the third affordance control context-specific audio settings. For example, the second affordance controls a volume for applications (e.g., application sounds, audio for media corresponding to an application, and/or application notifications), and the third affordance controls a volume for communications (e.g., phone calls, video calls, and/or AR/VR communication sessions with other users). In some embodiments, at the second time, the one or more affordances for accessing the first set of functions of the first computer also includes a fourth affordance (e.g., in addition to the second affordance and the third affordance). For example, the fourth affordance controls a volume for experiences (e.g., an AR or a VR experience). In some embodiments, at the first time, the first affordance controls the same context-specific settings, but controls only the respective setting for a respective setting (e.g., if an application user interface is displayed, the first affordance controls the volume for applications; if a communication session is active, the first affordance controls the volume for communication session; and while an AR or VR experience is active, the first affordance controls the volume for experiences). At the second time, no relevant context is active (e.g., no applications are open, no communication sessions are active, and no AR or VR experiences are active), or multiple relevant contexts are active, and so the second affordance, third affordance, and/or fourth affordance are displayed (e.g., to allow the user to adjust the relevant settings without first needing to trigger the appropriate context, or without affecting other active context(s)). In some embodiments, the first affordance controls more than one of the same context-specific settings (e.g., an application user interface is displayed and a communication session is active, so the first affordance controls the volume for both applications and communications (e.g., without controlling the volume for experiences)). In some embodiments, the first affordance controls all of the context-specific settings (e.g., the user can adjust the volume across multiple contexts without needing to individually adjust the second, third, and/or fourth affordances). For example, as described with reference to
In some embodiments, while displaying the third user interface object (e.g., the indicator 7010 of system function menu of
In some embodiments, the third user interface object (e.g., the indicator 7010 of system function menu of
In some embodiments, while the three-dimensional environment is visible through the viewport, the computer system displays the third user interface object with a first appearance at a first indicator position in the three-dimensional environment, wherein the first appearance of the third user interface object at the first indicator position is based at least in part on a characteristic of the three-dimensional environment at the first indicator position in the viewport through which the three-dimensional environment is visible. In response to detecting a movement of the viewpoint of the user from the first viewpoint to the third viewpoint in the three-dimensional environment, the computer system displays the third user interface object with a respective appearance at a respective indicator position in the three-dimensional environment that has the first spatial relationship to the viewport through which the three-dimensional environment is visible, wherein the respective appearance of the first user interface object at the respective indicator position is based at least in part on a characteristic of the three-dimensional environment at the respective indicator position. For example, as described above with reference to
In some embodiments, displaying the first user interface object in response to detecting the first user input including the first gaze input, includes displaying an animated transition of the one or more affordances for accessing the first set of functions of the first computer system emerging from the third user interface object in a first direction (e.g., the system function menu 7024 expanding downward, upward, leftward, rightward from the indicator, and/or or expanding from a peripheral region of the field of view toward a central or interior region of the field of view, in
In some embodiments, displaying the first user interface object in response to detecting the first user input including the first gaze input, includes displaying an animated transition of the one or more affordances for accessing the first set of functions of the first computer system gradually appearing. This is described above, for example, in the description of
In some embodiments, in response to detecting the first user input that includes the first gaze input, and in accordance with a determination that the first position in the three-dimensional environment has the first spatial relationship to the viewport through which the three-dimensional environment is visible (e.g., a determination that the first position is represented in a first region of the field of view provided by the first display generation component (e.g., in the upper left corner, top center region, lower right corner, peripheral region, or other preselected region of the field of view), while the viewport through which the three-dimensional environment is visible), the computer system displays an indication of the first user interface object (e.g., the indication is a smaller and/or more translucent version of the first user interface object, the indication is the indicator 7010 of system function menu of
In some embodiments, the first position in the three-dimensional environment is in a periphery region of the viewport through which the three-dimensional environment is visible. For example, in
In some embodiments, while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, detecting a fifth user input including detecting gaze input directed to a respective affordance of the one or more affordances. In response to detecting the fifth user input: in accordance with a determination that the respective affordance is a first affordance corresponding to a first unction of the first computer system and that the fifth user input includes a gesture input that meets gesture criteria, the computer system performs the first function; in accordance with a determination that the respective affordance is the first affordance corresponding to the first function of the first computer system and that the fifth user input does not include a gesture input that meets the gesture criteria, the computer system forgoes performing the first function; and in accordance with a determination that the respective affordance is a second affordance corresponding to a second function of the first computer system and that the fifth user input does not include a gesture input that meets the gesture criteria, the computer system performs the second function. For example, in some embodiments, some affordances in the first user interface object require the user to gaze and use a hand gesture to activate, while others can be activated with gaze without requiring a corresponding gesture. For example, in some embodiments, affordances for opening the notification center, adjusting a volume, opening the control center, opening the settings, require a gaze input as well as a corresponding gesture to activate; while affordances for the home, contacts, and other functions are activated by gaze without a gesture. For example, as described with reference to
In some embodiments, while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, the computer system detects a change in pose of a first portion of the user (e.g., the user’s hand, the user’s fingers, or other portions of the user). In response to detecting the change in pose of the first portion of the user: in accordance with a determination that the change in pose of the first portion of the user results in a first type of pose (e.g., a ready state pose, raised, forming a pinch, thumb on the side of the index finger, and/or other ready state poses), the computer system changes an appearance of the respective affordance (e.g., highlighting (e.g., making it larger, moving it toward the user, making it brighter, and/or otherwise enhancing the visibility thereof) the respective affordance relative to its surroundings, including the environment and/or other affordances in the first user interface object); and in accordance with a determination that the change in pose of the first portion of the user does not result in the first type of pose, the computer system forgoes changing the appearance of the respective affordance. For example, as described with reference to
In some embodiments, in response to detecting the change in pose of the first portion of the user, and in accordance with a determination that the change in pose of the first portion of the user results in the first type of pose (e.g., a ready state pose, raised, forming a pinch, thumb on the side of the index finger, and/or other ready state poses), the computer system forgoes changing an appearance of at least one affordance of the one or more affordances different from the respective affordance. For example, in
In some embodiments, while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, the computer system detects, via the one or more input devices, a sixth user input including gaze input directed to a respective affordance of the one or more affordances. In response to detecting the sixth user input directed to the respective affordance, the computer system displays (e.g., optionally, in accordance with a determination that the gaze input corresponding to the sixth user input has been maintained on the respective affordance for at least the first threshold amount of time) additional content associated with the respective affordance (e.g., a glyph, icon, and/or text showing the function that would be performed when the respective affordance is activated (e.g., with gaze and a gesture, and/or with a gaze and dwell)). This is described above, for example, with reference to
In some embodiments, while displaying the first user interface object, the computer system detects, via the one or more input devices, a seventh user input that activates a first affordance of the one or more affordances for accessing the first set of functions of the first computer system. In response to detecting the seventh user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) that activates the first affordance, the computer system displays a first system user interface for a first system function of the first computer system (e.g., a home user interface, a control user interface, a notification user interface, a settings user interface, and/or other types of system spaces) in the three-dimensional environment (e.g., in the center of the field of view, or in another portion of the field of view). For example, in
In some embodiments, while displaying the first user interface object and the first system user interface, the computer system detects, via the one or more input devices, an eighth user input that activates a second affordance, different from the first affordance, of the one or more of affordances for accessing the first set of functions of the first computer system. In response to detecting the eighth user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) that activates the second affordance, the computer system displays a second system user interface, different from the first system user interface, for a second system function of the first computer system, and the computer system ceases to display the first system user interface (e.g., replacing the first system user interface with the second system user interface at the display location of the first system user interface). This is described above, for example, in the description of
In some embodiments, while displaying the first system user interface and the first user interface object, the computer system detects, via the one or more input devices, a ninth user input that includes a gaze input (e.g., optionally in combination with an air gesture (e.g., an air tap or an air pinch), a touch gesture, an input provided via a controller a voice command, and/or another type of input) directed to the first user interface object (e.g., that, optionally, activates a respective affordance after a dwell time). In response to detecting the ninth user input, the computer system changes one or more visual properties (e.g., blurring, dimming, darkening, reducing opacity and/or color saturation) of the first system user interface to reducing visual prominence of the first system user interface relative to the first user interface object. In some embodiments, changing the one or more visual properties of the first system user interface include reducing visual prominence of the first system user interface from a steady state appearance of the first system user interface. In some embodiments, changing the one or more visual properties of the first system user interface is accompanied by changing one or more visual properties of the first user interface object to increase the visual prominence of the first user interface object relative to its previous visual appearance (e.g., a steady state appearance or a visually obscured appearance). This is described above, for example, in the description of
In some embodiments, while displaying the first system user interface and the first user interface object, the computer system detects, via the one or more input devices, a tenth user input that includes gaze input (e.g., optionally in combination with an air gesture (e.g., an air tap or an air pinch), a touch gesture, an input provided via a controller a voice command, and/or another type of input) directed to the first system user interface (e.g., the nine gaze input moved from the first user interface object or another portion of the three-dimensional environment to the first system user interface).In response to detecting the tenth user input, the computer system changes one or more visual properties (e.g., blurring, dimming, darkening, reducing opacity and/or color saturation) of the first user interface object to reduce visual prominence of the first user interface object relative to the first system user interface. In some embodiments, changing the one or more visual properties of the first user interface object include reducing visual prominence of the first user interface object from a steady state appearance of the first user interface object. In some embodiments, changing the one or more visual properties of the first user interface object is accompanied by changing one or more visual properties of the first system user interface to increase the visual prominence of the first system user interface relative to its previous visual appearance (e.g., a steady state appearance or a visually obscured appearance). This is described above, for example, in the description of
In some embodiments, while displaying, via the first display generation component, an application launching user interface in the three-dimensional environment (e.g., a home user interface that includes a plurality of application icons (e.g., application icons for a messaging application, an email application, a meditation application, a video chat application, a word processing application, a gaming application, a browser application, and other user and system applications), which when selected, causes the first computer system to display corresponding applications in the three-dimensional environment) and the first user interface object, the computer system detects, via the one or more input devices, an eleventh user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) that activates a third affordance of the one or more of affordances for accessing the first set of functions of the first computer system. In response to detecting the eleventh user input that activates the respective affordance, the computer system displays a third system user interface for a third system function of the first computer system that corresponds to the third affordance, and the computer system ceases to display the application launching user interface (e.g., replacing the application launching user interface with the third system user interface at the display location of the third system user interface). For example, as described with reference to
In some embodiments, while displaying, via the first display generation component, an application user interface in the three-dimensional environment (e.g., a user interface for a messaging application, an email application, a meditation application, a video chat application, a word processing application, a gaming application, a browser application, and other user applications, and optionally, system applications), which when selected, causes the first computer system to display corresponding applications in the three-dimensional environment) and the first user interface object, the computer system detects, via the one or more input devices, a twelfth user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) that activates a fourth affordance of the one or more of affordances for accessing the first set of functions of the first computer system. In response to detecting the twelfth user input that activates the respective affordance, the computer system displays a fourth system user interface for a fourth system function of the first computer system that corresponds to the fourth affordance, concurrently with the application user interface (e.g., overlaying or displayed side-by-side with the application user interface). For example, as described with reference to
In some embodiments, while displaying, via the first display generation component, the first user interface object, the computer system detects, via the one or more input devices, a thirteenth user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) that activates a fifth affordance of the one or more of affordances for accessing the first set of functions of the first computer system. In response to detecting the twelfth user input that activates the fifth affordance, the computer system performs a respective operation that corresponds to activation of the fifth affordance (e.g., changing the volume, display brightness, level of immersion, and other settings of the first computer system and/or the three-dimensional environment). For example, in
In some embodiments, performing the respective operation includes displaying one or more controls for adjusting one or more settings of the first computer system (e.g., the fifth affordance expands to display the one or more controls), wherein the one or more controls are displayed overlaying at least a portion of the first user interface object (e.g., overlaying and visually obscuring one or more of the plurality of affordances other than the fifth affordance). For example, as described above with respect to
In some embodiments, detecting the thirteenth user input that activates the fifth affordance of the one or more of affordances for accessing the first set of functions of the first computer system includes detecting a pinch and release gesture (e.g., two fingers of a hand coming into contact with each other, and then moving apart from each other after less than a threshold amount of time remaining in contact with each other) that is directed to the fifth affordance (e.g., with a gaze input directed to or a cursor pointed at the fifth affordance, or a pinch gesture directly occurring at the location of the fifth affordance). While displaying the one or more controls for adjusting one or more settings of the first computer system (e.g., after detecting the release portion of the pinch and release gesture, and before a threshold amount of time has elapsed since the pinch and release gesture), the computer system detects a pinch and drag gesture (e.g., two fingers of a hand coming into contact with each other, and then moving together in a respective direction while remaining in contact with each other) that is directed to a first control of the one or more controls (e.g., with a gaze input directed to or a cursor pointed at the first control, or a pinch gesture directly occurring at the location of the first control). In response to detecting the pinch and drag gesture that is directed to the first control of the one or more controls, adjusting a first setting that corresponds to the first control in accordance with one or more characteristics of the pinch and drag gesture (e.g., the location, speed, movement direction, and/or duration of the pinch and drag gesture). For example, as described with reference to
In some embodiments, displaying the first user interface object in the first view of the three-dimensional environment includes displaying the first user interface object at a first simulated distance from the first viewpoint of the user, wherein the first simulated distance is less than respective simulated distances of one or more other user interface objects displayed in the first view of the three-dimensional environment (e.g., one or more application user interfaces, one or more windows, and/or other user interface objects currently displayed in the three-dimensional environment) from the first viewpoint of the user. For example, in
In some embodiments, the computer system displays a plurality of system status indicators that include information about a status of the first computer system, concurrently with displaying the first user interface object. In some embodiments, the system status indicators and the one or more affordances for accessing the set of functions of the first computer system are displayed in the same container object (e.g., the first user interface object). In some embodiments, the system status indicators and the one or more affordances for accessing the set of functions of the first computer system are displayed in separate container objects (e.g., the first user interface object does not include the status indicators). For example, as described with reference to
In some embodiments, the first user interface object is displayed while the first gaze input remains within a first level of proximity to the first user interface object. While displaying the first user interface object, the computer system detects that the first gaze input moves away from the first user interface object. In response to detecting that the first gaze input has moved away from the first user interface object (e.g., the user has looked down toward the center of the field of view, or moved the gaze to another object in the field of view): in accordance with a determination that the first gaze input is beyond the first level of proximity to the first user interface object, the computer system ceases to display the first user interface object; and in accordance with a determination that the first gaze input remains within the first level of proximity to the first user interface object, the computer system maintains display the first user interface object. For example, as described with reference to
In some embodiments, after ceasing to display the first user interface object, the computer system detects that the first gaze input has moved back to the first position that has the first spatial relationship to the viewport through which the three-dimensional environment is visible. In response to detecting that the first gaze input has moved back to the first position, the computer system redisplays the first user interface object. For example, as described with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 12000 in some circumstances has a different appearance as described in the methods 9000-11000, and 13000-16000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-11000, and 13000-16000). For brevity, these details are not repeated here.
Displaying a first set of one or more user interface objects that correspond to a first set of one or more contextual conditions, and displaying a second set of one or more user interface objects that correspond to a second set of one or more contextual conditions, in response to detecting that the attention of the user is directed to a first portion of the first view of the three-dimensional environment, enables the computer system to automatically display different content (e.g., sets of one or more user interface objects) based on what contextual conditions are met at the time a user’s attention is directed to a portion of the first view of the three-dimensional environment. This reduces the number of user inputs needed to display contextually relevant content (e.g., the user does not need to manually navigate to contextually relevant content, and/or the user does not need to perform additional user inputs to select what content is (or is not) contextually relevant at the current time).
While a first view of a three-dimensional environment is visible via the first display generation component, the computer system detects (13002), via the one or more input devices, that attention of a user of the computer system is directed to a first portion of the first view of the three-dimensional environment (e.g., detecting a first gaze input that is directed to a first position in the three-dimensional environment; detecting a focus selector that is moved to the first position in the three-dimensional environment, and/or detecting other user inputs that indicate that the attention of the user has moved to the first position in the three-dimensional environment) (e.g., the user’s attention 7116 in
In some embodiments, while the first set of user interface object and the second set of user interface objects are not visible in a currently displayed view (e.g., the first view, or a second view displayed after the first view (e.g., as a result of the movement of the first display generation component relative to a physical environment surrounding the first display generation component, and/or as a result of movement of the viewpoint of the user)) of the three-dimensional environment (e.g., optionally, while the system function menu 7024 described in
For example, as described with reference to
In some embodiments, in response to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, and in accordance with a determination that the first portion of the first view of the three-dimensional environment does not have the first spatial relationship to the viewport through which the three-dimensional environment is visible (e.g., the first portion of the first view is in a second region (e.g., outside of the peripheral region, outside of a top center region, outside of an upper left corner region, and/or outside of another reactive region) of the first view that is not reactive to user attention and/or that is not associated with displaying the system function menu 7024 described in
For example, as described with reference to
In some embodiments, while displaying the first set of one or more user interface objects (e.g., that has been displayed in response to detecting the user’s attention being directed to a portion of the currently displayed view that has the first spatial relationship to the currently displayed view (and/or to the viewport through which the three-dimensional environment is visible) while the first set of contextual conditions are met), the computer system detects a first user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) directed to a respective user interface object of the first set of one or more user interface object (e.g., the respective user interface object is a notification, an alert, an update for a subscribed activity, and/or other content or control relevant to the first set of contextual conditions). In response to detecting the first user input, the computer system performs a first operation corresponding to the respective user interface object. In some embodiments, while displaying another set of one or more user interface objects that has been displayed in response to detecting the user’s attention being directed to a portion of the currently displayed view that has the first spatial relationship to the currently displayed view (and/or to the viewport) while another set of contextual conditions are met, the computer system performs a different operation in response to a user input directed to a respective user interface object of the displayed set of user interface objects. In some embodiments, allowing different sets of user interface objects to be displayed based on the current context enables different operations to be available to be performed depending on the context. For example, some possible contexts include an event corresponding to a time-sensitive notification (e.g., marked urgent, determined as urgent based on its content, or determined as time-sensitive based on keywords in its content and/or how it has been sent (e.g., with a time-sensitive flag, repeated without a short period of time, and/or other conditions)), a current time being within a threshold time period before a scheduled flight (e.g., based on a boarding pass stored on the computer system), ongoing media playback, and a live event with time-sensitive updates (e.g., food delivery in progress, sports game that is in progress, a detected workout that is in progress). In some embodiments, if the context involves an urgent notification, the operation includes responding to the notification, dismissing the notification, and/or opening an application associated with the notification. In some embodiments, if the context involves an alert or request that requires the user’s input to be dismissed, the operation includes dismissing the alert, providing an affordance for receiving the user’s input, and/or providing an affordance to turn off the alert and/or changing the delivery mode of similar alerts and requests of the current alert and/or request. In some embodiments, if the context involves a scheduled flight or boarding pass, the operation includes displaying the corresponding boarding pass, and/or displaying flight information. In some embodiments, if the context is ongoing media playback, the operation includes pausing the media playback, rewind or fast forwarding, navigating between different available tracks, and/or adjusting a volume of the media playback. In some embodiments, if the context is food delivery in progress, the first operation includes displaying information regarding the delivery (e.g., estimated delivery time), contacting a delivery driver, and/or requesting customer support. In some embodiments, if the context is a live sports game with time-sensitive updates, the first operation includes displaying a score and/or displaying player information or statistics. In some embodiments, if the context is a detected workout in progress, the first operation includes displaying information corresponding to the detected workout (e.g., calories burned and/or other biometric data received from one or more sensors in communication with the computer system). For example, in
In some embodiments, performing, in response to detecting the first user input, the first operation corresponding to the respective user interface object includes: in accordance with a determination that the first user input is a first type of input and the respective user interface object includes first information (e.g., is a first type of notification, a second type of notification, a first type of alert, a second type of alert, a live activity of a first type, a live activity of a second type, a session of a first type, a session of a second type, a first type of system status, a second type of system status, a first type of system control, or a second type of system control), performing a respective operation corresponding to the first information; and in accordance with a determination that the first user input is the first type of input and the respective user interface object includes second information (e.g., is a first type of notification, a second type of notification, a first type of alert, a second type of alert, a live activity of a first type, a live activity of a second type, a session of a first type, a session of a second type, a first type of system status, a second type of system status, a first type of system control, or a second type of system control) different from the first information, performing a respective operation corresponding to the second information, wherein the respective operation corresponding to the second information is different from the respective operation corresponding to the first information. In some embodiments, the respective user interface object including the first information is a notification, and the respective operation corresponding to the first information includes opening a first application associated with the notification and displaying content of the notification in the first application, or presenting a reply user interface (e.g., in the first application) for responding to the notification. In some embodiments, the respective user interface object including the second information is an alert, and the respective operation corresponding to the second information includes opening a modal window displaying more details associated with the alert, and optionally, displaying one or more controls to dispose of the alert in different ways (e.g., dismiss, snooze, or disabling similar types of alerts for the future). In some embodiments, the respective user interface object includes a respective type of information includes information regarding a subscribed live activity (e.g., delivery, sports game, flight information, news feed, and/or other activities with time-sensitive updates), and the first operation corresponds to the type of subscribed live activity (e.g., providing estimated arrival time and/or means to contact the driver, getting playback of scoring moments and/or getting player statistics, providing checking-in options and suggested transportation means, displaying related news or subscribe additional news feeds, and/or other operations corresponding to the type of information and the content of information presented in the respective user interface object). In some embodiments, the respective user interface object includes a respective type of information includes information regarding a system status or system control (e.g., battery level, connection type, network connection status, music playback status and controls, volume status and control, mobile carrier status, DND mode (e.g., status and/or control), fight mode (e.g., status and/or control), silent mode (e.g., status and/or control), and/or other system status and system controls), and the first operation corresponds to viewing and changing the system status or adjusting the system controls (e.g., changing battery charging mode, changing operation mode based on battery status, changing network, media playback, volume, and/or other system operation modes). For example, as described with reference to
In some embodiments, the first user input that is the first type of input includes a selection user input (e.g., a verbal input and/or an air gesture, such as an air tap or an air pinch) while the attention of the user is directed to the respective user interface object. For example, as described with reference to
In some embodiments, performing the first operation corresponding to the respective user interface object includes, in accordance with a determination that the respective user interface object is a first notification, displaying additional information corresponding to first notification (e.g., opening an application associated with the first notification and displaying the notification content in the context of other content and functions in the application). In some embodiments, in addition to displaying the additional information corresponding to the first notification, the computer system ceases to display other user interface objects (e.g., other user interface objects that also correspond to the first set of contextual conditions, or user interface objects that correspond to other contextual conditions that are also present) that were displayed with the respective user interface objects. For example, as described with reference to
In some embodiments, performing, in response to detecting the first user input, the first operation corresponding to the respective user interface object includes: in accordance with a determination that the first user input is a user input of a second type (e.g., the user’s attention (e.g., as indicated by the user’s gaze, or the position of a user controlled focus selector) being directed to the system function menu 7024 described with reference to
In some embodiments, the first user input that is a user input of the second type include a gaze input directed to a first region of the first view of the three-dimensional environment (e.g., the first region of the first view has a first spatial relationship to the first viewpoint of the user, or the indicator 7010 of system function menu (e.g., as described above with reference to
The method of any of claims 47-51, wherein performing, in response to detecting the first user input, the first operation corresponding to the respective user interface object includes: in accordance with a determination that the first user input is a user input of a third type (e.g., different from the user input of the first type and the user input of the second type), displaying a first additional user interface object (e.g., or a plurality of additional user interface objects) different from the respective user interface object. In some embodiments, the user input of the third type is a scroll input, and additional user interface objects are scrolled into view in response to the user input of the third type. In some embodiments, the additional user interface objects are other objects that correspond to the current context (e.g., context that met the first set of context conditions, context that met the second set of contextual conditions, or other context meeting other sets of context conditions). In some embodiments, the respective user interface object is a first notification of a plurality of stacked notifications, and a user input of the third type directed to the respective user interface object expands the stacked notifications and displays one or more additional notification with the first notification. In some embodiments, the respective user interface object is a representative object of a plurality of similar objects arranged in a stack, and a user input of the third type directed to the respective user interface object expands the stack and displays one or more additional objects of the same type or related types as the respective user interface object. For example, in
In some embodiments, displaying the first additional user interface object includes concurrently displaying the first additional user interface object with the respective user interface object (and, optionally, other user interface objects in the first set or second set of user interface objects that were displayed at the time when the user’s attention is directed to the first portion of the first view of the three-dimensional environment). For example, in
In some embodiments, displaying the first additional user interface object includes: in accordance with a determination that a total number of the first set of user interface objects (e.g., of the first or second set of one or more user interface objects) that are currently visible in the first view of the three-dimensional environment (e.g., those that were initially displayed in response to detecting the user’s attention being directed to the first portion of the first view) is fewer than a first threshold number, displaying the first additional user interface object while maintaining display of the first set of user interface objects in the first view of the three-dimensional environment; and in accordance with a determination that the total number of the first set of user interface objects (e.g., of the first or second set of one or more user interface objects) that are currently visible in the first view of the three-dimensional environment (e.g., those that were initially displayed in response to detecting the user’s attention being directed to the first portion of the first view) is greater than or equal to the first threshold number, displaying the first additional user interface object and removing display of at least one of the first set of user interface objects from the first view of the three-dimensional environment. In some embodiments, display of the user interface objects that correspond to the first set of contextual conditions is capped, such that no more than a threshold number (e.g., the first threshold number plus one) of user interface objects that correspond to the first set of contextual conditions are visible at a given time. If more objects are available for display, when an additional object is brought into view by a user input of the third type, another object that were displayed at the time of the user input ceases to be displayed (e.g., faded out partially or completely). For example, In
In some embodiments, displaying the first additional user interface object includes replacing display of the respective user interface object with the first additional user interface object. For example, as described with reference to
In some embodiments, at a first time when the respective user interface object occupies a respective portion of the first view that is associated with user attention (e.g., a central portion of the field of view, a portion that has the user’s gaze or focus selector, and/or a portion that is highlighted or selected): the respective user interface object is displayed at a first distance from a first viewpoint of the user that corresponds to the first view of the three-dimensional environment; the first additional user interface object is displayed at a second distance, different from the first distance (e.g., less than the first distance, or greater than the first distance), from the first viewpoint of the user at the first time; and at a second time when the first additional user interface object is moved into the respective portion of the first view that is associated with user attention (e.g., a central portion of the field of view, a portion that has the user’s gaze or focus selector, and/or a portion that is highlighted or selected), and the respective user interface is not displayed in the respective portion of the first view that is associated with user attention (e.g., the first additional user interface object is scrolled into, and the respective user interface object or another user interface object is scrolled out of the first portion of the first view):the first additional user interface object is displayed at the first distance from the first viewpoint of the user (e.g., optionally, the respective user interface object is displayed at the second distance, or a third distance, from the first viewpoint of the user). In some embodiments, the respective user interface object is displayed to appear further from the viewpoint of the user once the first additional user interface object moves into the region associated with higher user attention (e.g., because the computer system determines that the first additional user interface object has a higher priority, or is more relevant (e.g., is the target of the attention of the user) and should be displayed with increased prominence (e.g., closer to the viewpoint of the user)). In some embodiments, the newly displayed user interface object moves to a z-position that is closer to the current viewpoint of the user, to gain the user’s attention; and as the user causes additional user interface objects to be displayed (e.g., with additional inputs of the third input type), the previously displayed user interface objects recede or fade out of view, and the newly displayed user interface object is displayed at the same z-position from the current viewpoint of the user. For example, in
In some embodiments, the first user input that is the input of the third type includes a scrolling input (e.g.., including movement of in a first direction (e.g., an air pinch and drag gesture, a swipe gesture, optionally detected in conjunction with a gaze directed to the first set of user interface objects)). For example, as described with reference to
In some embodiments, the first user input that is the input of the third type includes a gaze input (e.g., a gaze input directed to a respective region for a threshold amount of time, a gaze and dwell input directed to a target object or location, optionally, without detecting another type of input (e.g., a gesture, a voice command, or another type of input modality)). In some embodiments, the computer system responds to gaze, optionally, with time, location, and/or stability criteria, to trigger display of the first additional user interface object (e.g., the system function menu 7024 of
In some embodiments, before displaying the first additional user interface object, the respective user interface object is displayed with a first level of visual prominence in the first view of the three-dimensional environment, and while displaying (e.g., with the first level of visual prominence, or another level of visual prominence higher than the first level of visual prominence) the first additional user interface object in the first view of the three-dimensional environment, the respective user interface object is displayed with a second level of visual prominence that is lower that the first level of prominence (e.g., the first set of user interface objects is displayed with reduced prominence, is dimmed, is blurred, or is no longer visible). While displaying the first additional user interface object, the computer system detects a second user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input). In response to detecting the second user input, in accordance with a determination that the second user input corresponds to a request to display the respective user interface object, the computer system displays the respective user interface object with the first level of prominence (e.g., displaying the respective user interface object with an original level of prominence) (and optionally, ceasing to display the first additional user interface object, or displaying the first additional user interface object with a reduced level of visual prominence). For example, in
In some embodiments, while displaying the first set of one or more user interface objects (e.g., that has been displayed in response to detecting the user’s attention being directed to a portion of the currently displayed view that has the first spatial relationship to the currently displayed view while the first set of contextual conditions are met), the computer system detects a third user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input). In response to detecting the third user input, and in accordance with a determination that the first user input is a user input of a fourth type (e.g., a swipe of the hand as a whole in the air, a tap on a close affordance associated with the first set of user interface objects) (e.g., different from the first type, second type, and third type), the computer system ceases to display the first set of one or more user interface objects (e.g., dismisses the set of user interface objects that is relevant to the first set of contextual conditions, without displaying or navigating to other user interface objects relevant to the first set of contextual conditions). In some embodiments, in accordance with a determination that the second set of user interface objects is displayed and in accordance with a determination that the first user input is a user input of the fourth type, the computer system ceases to display the second set of user interface objects (e.g., dismisses the set of user interface objects that is relevant to the second set of contextual conditions, without displaying or navigating to other user interface objects relevant to the second set of contextual conditions). In some embodiments, irrespective of which particular object in the first set of user interface object is the respective user interface object, the first set of user interface object as a whole is dismissed from the first view of the three-dimensional environment, in response to the user input of the fourth type. For example, as described with reference to
In some embodiments, the third user input that is an input of the fourth type includes a gaze input directed away from a region occupied by the first set of one or more user interface objects for at least a threshold amount of time (e.g., detecting the user’s attention being directed away from the system function menu 7024 of
In some embodiments, the third user input that is an input of the fourth type includes movement in a first direction (e.g., a scrolling movement with an air pinch and drag gesture in a first direction, a swipe gesture in a first direction, or other movement in a first direction that is caused by the user). In some embodiments, the third user input is a scroll input in an upward direction. In some embodiments, the third user input is a scroll input in a downward direction. In some embodiments, the third user input causes the additional user interface objects corresponding to the first set of contextual conditions to come into view, and causes the first set of user interface objects to pushed out of the first view. This is described with reference to
In some embodiments, while displaying the first set of one or more user interface objects in the first view of the computer-generated environment (e.g., or another view that corresponds to the current viewpoint of the user), the computer system detects occurrence of a first event. In response to detecting occurrence of the first event, the computer system adds a new user interface object (e.g., a new notification or alert, an update to a subscribed event, or other object relevant to the first event) to the first set of one or more user interface objects (e.g., displaying the new user interface object in the same arrangement as the first set of one or more user interface objects, optionally, pushing one or more of the existing object in the first set of user interface objects outside of the current view), wherein the new user interface object corresponds to the first event. For example, as described with reference to
In some embodiments, in response to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, and in accordance with a determination that the first portion of the first view has the first spatial relationship to the viewport through which the three-dimensional environment is visible, the computer system displays one or more indications of system status (e.g., a current date, a current time, a current battery level, a cellular signal strength, and/or wireless signal strength) for the computer system (e.g., regardless of which sets of contextual conditions are met, or whether any sets of contextual conditions are met at all). For example, as described with reference to
In some embodiments, while displaying the first set of one or more user interface objects, and while displaying the one or more indications of system status for the computer system, the computer system detects a fourth user input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input). In response to detecting the fourth user input, the computer system scrolls the first set of one or more user interface objects (e.g., ceasing to display at least one user interface object of the first set of one or more user interface objects, displaying at least one user interface object of the first set of one or more user interface objects with a different level of prominence, and/or displaying at least one user interface object that corresponds to the first set of one or more contextual conditions, but was not previously visible), while maintaining display of the one or more indications of system status. For example, as described with reference to
In some embodiments, while displaying the first set of user interface objects that correspond to the first set of contextual conditions, the computer system concurrently displays an indication of a second additional user interface object (e.g., an edge portion, a translucent and/or reduced representation, or some other indication) that corresponds to the first set of contextual conditions and that is available to be displayed (e.g., the second additional user interface object is partially occluded by the first set of one or more user interface objects). For example, in
In some embodiments, while the first set of one or more user interface objects is displayed with a first level of visual prominence (e.g., with a first distance from the viewpoint of the user, a first brightness, first opacity, and/or first level of blur), the indication of the second additional user interface object is displayed with a second level of visual prominence (e.g., with a second distance from the viewpoint of the user, a second brightness, second opacity, and/or second level of blur) that is lower than the first level of visual prominence (e.g., the first set of one or more user interface objects is displayed in front of or on top of (e.g., in the view of three-dimensional environment) the second additional user interface object, and/or the second additional user interface object appears dimmer, more translucent or transparent, and/or blurrier than the first set of one or more user interface objects). For example, in
In some embodiments, while the first set of one or more user interface objects is displayed with the first level of visual prominence (e.g., with a first distance from the viewpoint of the user, a first brightness, first opacity, and/or first level of blur) and the indication of the second additional user interface object is displayed with the second level of visual prominence (e.g., with a second distance from the viewpoint of the user, a second brightness, second opacity, and/or second level of blur) that is lower than the first level of prominence, the computer system detects a fifth input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) directed to the indication of the second additional user interface object (e.g., a gaze input, a gaze and dwell input, a gaze input in conjunction with an air gesture (e.g., an air tap, or air pinch gesture)). In response to detecting the fifth user input directed to the indication of the second additional user interface object, the computer system displays the second additional user interface object with a third level of visual prominence, wherein the third level of visual prominence is greater than the second level of prominence. In some embodiments, the third level of visual prominence is the same as the first level of visual prominence. For example, in
In some embodiments, displaying the first set of one or more user interface objects includes displaying content (e.g., user interface objects including information related to the current state and/or controls for modifying the current state) corresponding to (e.g., conveying the status, change in status, and/or other information of) the current state of the computer system (e.g., one or more ongoing sessions, such as real-time communication sessions (e.g., telephone calls, video conferences, and/or copresence or shared experiences in an AR or VR environment), vehicle navigation sessions, active guest mode, full-screen media playback mode, screen recording mode, screen mirroring mode, and/or other active and ongoing events or operation modes of the computer system). For example, as described with reference to
In some embodiments, displaying the first set of one or more user interface objects includes displaying a first set of one or more affordances for performing operations corresponding to the first set of one or more contextual conditions (e.g., affordances for accepting/declining a real-time communication session, terminating or modifying a vehicle navigation session, ending or modifying an active guest mode, existing or pausing a full-screen media playback mode, ending or pausing a screen recording mode, ending, pausing, or reconfiguring a screen mirroring mode, and/or affordance for otherwise changing or terminating other active and ongoing events or operation modes). For example, as described with reference to
In some embodiments, displaying the first set of one or more user interface objects includes displaying application content corresponding to a first application (e.g., content such as notifications, file transfers, and download progress corresponding to respective applications). For example, as described with reference to
In some embodiments, the first set of one more user interface object includes two or more of: a first user interface object that includes first application content from the first application (e.g., notifications from the first application, file transfer for a file type associated with the first application, download progress for downloading the first application, and/or other content from the first application); a second user interface object that includes second application content from a second application different from the first application (e.g., notifications from the second application, file transfer for a file type associated with the second application, download progress for downloading the second application, and/or other content from the second application); and a third user interface object that includes system content (e.g., system alerts, system status indicators, and other system-related content that is not specific to a particular application installed on the system) corresponding to a current state of the computer system (e.g., charging status, low battery status, screen recording, privacy mode, DND mode, file sharing, file transfer, and network connectivity, peripheral device connectivity, and other system status and operation modes). For example, as described with reference to
In some embodiments, displaying the first set of one or more user interface objects includes displaying the first set of one or more user interface objects with first prioritization of the first set of one or more user interface objects that is based on the first set of one or more contextual conditions, and displaying the second set of one or more user interface objects includes displaying the second set of one or more user interface objects with second prioritization of the second set of one or more user interface objects that is based on the second set of one or more contextual conditions. In other words, the user interface objects in the contextual user interface are prioritized based on a state of the computer system at a time when the contextual user interface was invoked. User interface objects can be prioritized in different ways. In some embodiments, prioritization includes an order of user interface objects (e.g., objects with higher priority are displayed in front of and/or on top of objects with lower priority) (e.g., objects are displayed in a list (e.g., with the same z-depth), and objects ranked higher in the list (e.g., towards the top of the list) are prioritized over objects that are lower in the list. In some embodiments, prioritization includes whether an object is displayed or not (e.g., objects that are prioritized are displayed, while objects that are not prioritized are not displayed). In some embodiments, prioritization includes relative visual prominences of the displayed objects (e.g., objects with higher priority are displayed with higher levels of visual prominence (e.g., brighter, with a different color, with thicker borders, with a larger size, with larger text, with bolded text, and/or with a different font) as compared to objects with lower priority, which are displayed with lower levels of visual prominence (e.g., dimmer, with thinner (or default sized) borders, with a smaller (or default) size, with smaller (or default) text, without bolded text, and/or with a default font). For example, as described with reference to
In some embodiments, prior to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, the computer system displays a preview (e.g., an icon, an indicator, or other types of preview or hint) of a first user interface object of the first set of user interface objects (e.g., a first communication request or a first notification) (e.g., in the first portion of the first view of the three-dimensional environment, in the system function menu 7024 described with reference to
In some embodiments, in accordance with a determination that the first set of user interface objects includes the first user interface object that corresponds to a first notification and includes a second user interface object that corresponds to a first request, the computer system prioritizes display of the second user interface object over display of the first user interface object (when displaying the first set of user interface objects in response to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment). User interface objects can be prioritized in different ways. In some embodiments, prioritization includes an order of user interface objects (e.g., objects with higher priority are displayed in front of and/or on top of objects with lower priority) (e.g., objects are displayed in a list (e.g., with the same z-depth), and objects ranked higher in the list (e.g., towards the top of the list) are prioritized over objects that are lower in the list. In some embodiments, prioritization includes whether an object is displayed or not (e.g., objects that are prioritized are displayed, while objects that are not prioritized are not displayed). In some embodiments, prioritization includes relative visual prominences of the displayed objects (e.g., objects with higher priority are displayed with higher levels of visual prominence (e.g., brighter, with a different color, with thicker borders, with a larger size, with larger text, with bolded text, and/or with a different font) as compared to objects with lower priority, which are displayed with lower levels of visual prominence (e.g., dimmer, with thinner (or default sized) borders, with a smaller (or default) size, with smaller (or default) text, without bolded text, and/or with a default font). For example, in
In some embodiments, prior to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment, the computer system displays a preview (e.g., an icon, an indicator, or other types of preview or hint) of a third user interface object of the first set of user interface objects (e.g., a first activity) (e.g., in the first portion of the first view of the three-dimensional environment, in the system function menu 7024 described with reference to
In some embodiments, in accordance with a determination that no preview of a respective user interface object of the first set of user interface objects was displayed before (e.g., at the time that, and/or within a first time window of) the attention of the user is directed to the first portion of the first view of the three-dimensional environment, the computer system prioritizes display of a fourth user interface object that corresponds to a system control (e.g., volume control, display brightness control, network connectivity control, media playback controls, and other system controls) in the first set of user interface objects (e.g., highlighting the fourth user interface object; displaying the fourth user interface object in a center or top of the first set of user interface objects, when displaying the first set of user interface objects in response to detecting that the attention of the user is directed to the first portion of the first view of the three-dimensional environment; and/or displaying the first user interface object and not displaying at least some other objects in the first set of user interface objects). User interface objects can be prioritized in different ways. In some embodiments, prioritization includes an order of user interface objects (e.g., objects with higher priority are displayed in front of and/or on top of objects with lower priority) (e.g., objects are displayed in a list (e.g., with the same z-depth), and objects ranked higher in the list (e.g., towards the top of the list) are prioritized over objects that are lower in the list. In some embodiments, prioritization includes whether an object is displayed or not (e.g., objects that are prioritized are displayed, while objects that are not prioritized are not displayed). In some embodiments, prioritization includes relative visual prominences of the displayed objects (e.g., objects with higher priority are displayed with higher levels of visual prominence (e.g., brighter, with a different color, with thicker borders, with a larger size, with larger text, with bolded text, and/or with a different font) as compared to objects with lower priority, which are displayed with lower levels of visual prominence (e.g., dimmer, with thinner (or default sized) borders, with a smaller (or default) size, with smaller (or default) text, without bolded text, and/or with a default font). For example, in
In some embodiments, in accordance with a determination that the first set of user interface objects include two or more user interface objects that correspond to notifications (and/or requests), the computer system prioritizes display of the two or more user interface objects that correspond to notification (and/or requests) in accordance with respective timestamps associated with the two or more user interface objects (e.g., listing them in chronological order or reverse chronological order). For example, for user older notifications can be assigned lower priority scores, such that the most recent notifications are displayed with the most prominence, while older notifications are displayed with reduced prominence. In some embodiments, notifications are assigned a second priority score (or a sub-priority score), for purposes of ordering notifications with respect to other notifications, while a (first) priority score (or actual priority score) determines how notification user interface objects (collectively) are ordered with respect to user interface objects of other types (e.g., user interface objects corresponding to communication requests, activities, or a plurality of affordances for accessing a first set of functions of the computer system). User interface objects can be prioritized in different ways. In some embodiments, prioritization includes an order of user interface objects (e.g., objects with higher priority are displayed in front of and/or on top of objects with lower priority) (e.g., objects are displayed in a list (e.g., with the same z-depth), and objects ranked higher in the list (e.g., towards the top of the list) are prioritized over objects that are lower in the list. In some embodiments, prioritization includes whether an object is displayed or not (e.g., objects that are prioritized are displayed, while objects that are not prioritized are not displayed). In some embodiments, prioritization includes relative visual prominences of the displayed objects (e.g., objects with higher priority are displayed with higher levels of visual prominence (e.g., brighter, with a different color, with thicker borders, with a larger size, with larger text, with bolded text, and/or with a different font) as compared to objects with lower priority, which are displayed with lower levels of visual prominence (e.g., dimmer, with thinner (or default sized) borders, with a smaller (or default) size, with smaller (or default) text, without bolded text, and/or with a default font). For example, as described with reference to
In some embodiments, in accordance with a determination that the first set of user interface objects include two or more user interface objects that correspond to requests, the computer system prioritizes display of the two or more user interface objects (e.g., displaying the two or more user interface objects above and/or on top of other user interface objects in the first set of user interface objects, displaying the two or more user interface objects while not displaying at least some other user interface objects in the first set of user interface objects, and/or displaying the two or more user interface objects with increased visual prominence relative to other user interface objects in the first set of user interface objects) that correspond to requests in accordance with preestablished prioritization for different types of requests (e.g., copresence requests are prioritized over video conference requests, and video conference requests are prioritized over voice-only communication requests). User interface objects can be prioritized in different ways. In some embodiments, prioritization includes an order of user interface objects (e.g., objects with higher priority are displayed in front of and/or on top of objects with lower priority) (e.g., objects are displayed in a list (e.g., with the same z-depth), and objects ranked higher in the list (e.g., towards the top of the list) are prioritized over objects that are lower in the list. In some embodiments, prioritization includes whether an object is displayed or not (e.g., objects that are prioritized are displayed, while objects that are not prioritized are not displayed). In some embodiments, prioritization includes relative visual prominences of the displayed objects (e.g., objects with higher priority are displayed with higher levels of visual prominence (e.g., brighter, with a different color, with thicker borders, with a larger size, with larger text, with bolded text, and/or with a different font) as compared to objects with lower priority, which are displayed with lower levels of visual prominence (e.g., dimmer, with thinner (or default sized) borders, with a smaller (or default) size, with smaller (or default) text, without bolded text, and/or with a default font). For example, as described with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 13000 in some circumstances has a different appearance as described in the methods 9000-12000, and 14000-16000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-12000, and 14000-16000). For brevity, these details are not repeated here.
Displaying a first indication of a first notification corresponding to a first event, and maintaining display of the first indication for a first duration of time and ceasing display of the first indication after expiration of the first duration of time, in accordance with a determination that attention of a user of the computer system was directed to the first indication within a first threshold amount of time and then moved away from the first indication before first criteria are met, and maintaining display of the first indication for a second duration of time and ceasing display of the first indication after expiration of the second duration of time different from the first duration of time, in accordance with a determination that the attention of the user of the computer system was not directed to the first indication within the first threshold amount of time, reduces the number of user inputs needed to cease display of the first indication after an appropriate amount of time (e.g., if the user’s attention is directed to the first indication within a first threshold amount of time, the first indication ceases to be displayed after a shorter amount of time (e.g., because the user’s attention was directed to the first indication and/or the user was able to view at least some content associated with the first notification); and if the user’s attention is not directed to the first indication within the first threshold amount of time, the first indication ceases to be displayed after a longer period amount of time (e.g., so that the user has time to direct the user’s attention to the first indication before the first indication ceases to be displayed)).
While a first view of an environment (e.g., a three-dimensional environment such as an AR, VR, or MR environment or a two-dimensional environment such as a user interface or desktop of a device) is visible via one or more display generation components, the computer system detects (14002) occurrence of a first event (e.g., a first event that triggers and/or causes display of a first notification). In response to detecting the occurrence of the first event, the computer system displays (14004), via the first display generation component, a first indication (e.g., an icon and/or a preview of at least a portion of notification content corresponding to the first notification) of a first notification corresponding to the first event (e.g., the user interface object 7056 in
In some embodiments, the second duration of time is longer than the first duration of time. For example, in
In some embodiments, the occurrence of the first event includes at least one of receiving an incoming communication request (e.g., a request for a telephone call, a VoIP call, a video conference call, or a copresence request for an AR or VR session), occurrence of a change in a state of an application (e.g., occurrence of an error, a request for user input, termination of a process, and/or other change in a state of the application), or occurrence of a change in a state of the computer system (e.g., charging completed, system update started or pending, low power mode started or ended, network connectivity started or interrupted, DND mode started or ended, and/or other changes in the system status or operation mode of the computer system). For example, as described with reference to
In some embodiments, after displaying, via the first display generation component, the first indication of the first notification corresponding to the first event: in accordance with a determination that the attention of the user of the computer system was directed to the first indication and that the first criteria (e.g., criteria for displaying the notification, or interacting with the notification content) are met (e.g., the computer system detects that the attention of the user is directed to the first indication for a threshold amount of time, before the computer system ceases display of the first indication after expiration of the first or second duration of time; or the computer system detects an air gesture (e.g., an air tap or an air pinch) while the attention of the user is directed to the first indication, before the computer system ceases display of the first indication after expiration of the first or second duration of time), the computer system displays the first notification including first notification content. In some embodiments, the first indication of the first notification does not include notification content for the first notification, and the first notification content for the first notification includes a first amount of notification content for the first notification, and the indication of the first notification includes no notification content or less notification content than the first notification content. In some embodiments, the first indication of the first notification includes some content (e.g., an application name, an application icon, a contact name, and/or a subject line or title corresponding to the notification content) that is displayed whether or not the attention of the user is directed to the first indication and the first criteria are met (e.g., some content is always displayed while the first indication of the first notification is displayed), but the first indication of the first notification does not (e.g., does not always) include all notification content (e.g., does not include the first notification content). For example, as described with reference to
In some embodiments, the first notification including the first notification content is concurrently displayed with one or more user interface objects that correspond to a first set of one or more contextual conditions that were met at a time when the attention of the user is directed to the first indication. In some embodiments, the first indication is displayed at a location that has the first spatial relationship to the first view of the environment (e.g., a reactive region for triggering display of the system function menu 7024 of
In some embodiments, while displaying the first notification including the first notification content, the computer system detects a first user input that meets second criteria with respect to the first notification (e.g., a tap input on the first notification, a tap and hold input, a gaze input that remains on the first notification for a threshold amount of time, a gaze input detected in conjunction with an air pinch or air tap gesture, or another type of user input that corresponds to a request to expand the first notification (e.g., as opposed to opening an application associated with the first notification)). In response to detecting the first user input that meets the second criteria, the computer system expands the first notification to display at least some of the first notification content and additional content (e.g., graphical, textual objects and/or control objects) corresponding to the first notification that was not included in the first notification content. In some embodiments, the first notification includes a first amount of notification content for the first notification before being expanded, and the expanded first notification includes a second amount, greater than the first amount, of notification content for the first notification. In some embodiments, the first notification content for the first notification is a notification preview that includes some, but not all, notification content for the first notification, and the expanded first notification includes the full notification content. For example, as described with reference to
In some embodiments, the second criteria include a first criterion (e.g., the first criterion is an only criterion of the second criteria, the first criterion is one of multiple joint criteria of the second criteria, or the first criterion is one of multiple alternative criteria of the second criteria) that requires the attention of the user to be (e.g., continuously) maintained on the first notification for at least a second threshold amount of time in order to be met. For example, as described with reference to
In some embodiments, the second criteria include a second criterion (e.g., the second criterion is an only criterion of the second criteria, the second criterion is one of multiple joint criteria of the second criteria, or the second criterion is one of multiple alternative criteria of the second criteria) that requires that a user input directed to the first notification is of a first input type (e.g., an air pinch, a long tap, or a double tap) in order to be met. For example, as described with reference to
In some embodiments, displaying the first indication of the first notification in response to detecting the occurrence of the first event includes displaying the first indication of the first notification at a first location in the environment, wherein the first location has a first spatial relationship to the first view of the environment. Prior to detecting the occurrence of the first event, the computer system detects that the attention of the user is directed to the first location (e.g., or a respective region that includes the first location) in first view of the environment. In response to detecting that the attention of the user is directed to the first location in the first view of the environment, the computer system displays a first user interface object at the first location in the first view of the environment. An example of the first user interface object is the indicator 7010 of system function menu described above (e.g., with reference to
In some embodiments, displaying the first indication of the first notification in response to detecting the occurrence of the first event includes displaying the first indication of the first notification at a first location in the first view of the environment. After ceasing display of the first indication (e.g., after expiration of the first duration of time or the second duration of time, and optionally, within a threshold amount of time of ceasing display of the first indication), the computer system detects that the attention of the user is directed to the first location. In response to detecting that the attention of the user is directed to the first location, the computer system displays a representation (e.g., e.g., the first indication, or the first notification that includes the first notification content) of the first notification (e.g., at the first location, or a location proximate to the first location). For example, as described with reference to
In some embodiments, in response to detecting that the attention of the user is directed to the first location, the computer system concurrently displays a first set of one or more user interface objects that correspond to a first set of one or more contextual conditions with the representation of the first notification (e.g., the representation of the first notification is the first notification that is displayed among other user interface objects (e.g., alerts, activities, and/or other notifications) in the first set of one or more user interface objects that correspond to the first set of one or more contextual conditions, or the representation of the first notification is the first indication of the first notification, and is redisplayed at the location of the indicator 7010 of system function menu (e.g., as shown in
In some embodiments, displaying the first indication of the first notification in response to detecting the occurrence of the first event includes displaying the first indication of the first notification at a first location in the three-dimensional environment. After ceasing display of the first indication of the first notification after expiration of the first duration of time or the second duration of time, the computer system detects that the attention of the user is directed to the first location. In response to detecting that the attention of the user is directed to the first location: in accordance with a determination that the attention of the user was directed to the first location within a third threshold amount of time (e.g., 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60 seconds, or another amount of time) of ceasing display of the first indication of the first notification, the computer system displays a representation (e.g., that includes notification content) of the first notification (e.g., the representation of the first notification is the first indication, or the first notification itself); and in accordance with a determination that the attention of the user was not directed to the first location within the third threshold amount of time of ceasing display of the first indication of the first notification, the computer system displays a second set of one or more user interface objects without displaying the representation of the first notification. In some embodiments, the second set of one or more user interface objects includes a plurality of affordances for accessing system functions of the computer system, such as the system function menu 7024; a contextually relevant user interface, such as the user interface 7068 for joining a communication session (e.g., if there is an active request to join a communication session; and/or a contextual user interface, as described above with reference to
In some embodiments, the second set of user interface objects includes one or more user interface objects that correspond to a respective set of one or more contextual conditions that are met at a time that the attention of the user was directed to the first location. For example, as described with reference to
In some embodiments, the second set of user interface objects includes one or more affordances for accessing system functions of the computer system (e.g., the second set of user interface objects include system status indicators and controls that are displayed in the system function menu 7024). For example, as described with reference to
In some embodiments, after ceasing display of the first indication of the first notification after expiration of the first duration of time or the second duration of time, the computer system detects a first request to display a notification user interface. In response to detecting the first request to display the notification user interface, the computer system displays the notification user interface, wherein the notification user interface includes a plurality of previously received notifications and the first notification. In some embodiments, the notification user interface includes the first notification even if the attention of the user was not directed to the first indication within the first threshold amount of time (e.g., and no additional content of the first notification, besides the first indication of the first notification, was ever displayed). For example, as described with reference to
In some embodiments, detecting the first request to display the notification user interface includes detecting user interaction with a system user interface that has been displayed in response to detecting that the attention of the user is directed to a respective region of the first view of the environment. For example, in some embodiments, the system user interface is the system function menu 7024 described with reference to
In some embodiments, the respective region of the first view of the environment is located at a first position in the environment that has a first spatial relationship to the first view of the environment. The computer system detects a change in a current viewpoint of the user from a first viewpoint corresponding to the first view of the environment, to a second viewpoint corresponding to a second view of the environment. In response to detecting the change in the current viewpoint of the user from the first viewpoint to the second viewpoint, the computer system updates a currently displayed view of the environment from the first view to the second view of the environment. While displaying the second view of the environment, the computer system detects that the attention of the user is directed to a respective region of the second view of the environment, wherein the respective region of the second view of the environment has the first spatial relationship to the second view of the environment. In some embodiments, the respective region of the first view of the environment is the same size as the respective region of the second view of the environment (e.g., the user can look to the same sized region in any respective view of the environment in order to trigger display of the notification user interface). In response to detecting that the attention of the user is directed to the respective region of the second view of the environment, the computer system displays the notification user interface (e.g., the system space 7052 that is a notification center in
In some embodiments, the first event is an event of a first type (e.g., events in applications that cause notifications to be generated by the applications, the applications including messaging applications, calendar application, email application, and other user applications that generate notifications, rather than temporary system alerts), and the first indication of the first notification corresponding to the first event is displayed at a first position in the environment (e.g., in a stack, below the system status indicators and indicator 7010 of system function menu (e.g., of
In some embodiments, the computer system ceases to display the second indication of the first alert corresponding to the second event (e.g., after expiration of a threshold amount of time that, optionally, is chosen based on whether the attention of the user has been detected on the second indication). After ceasing display of the second indication of the first alert corresponding to the second event, the computer system detects a second request (e.g., a gaze input directed to the notifications affordance 7044 in the system function menu 7024, as shown in
In some embodiments, while displaying the second indication of the first alert, the computer system detects that the attention of the user is directed to a respective region in the first view of the environment that includes a plurality of affordances for accessing system functions of the computer system (e.g., the user interface object 7056 in
In some embodiments, the second indication of the first alert includes one or more selectable options for performing operations corresponding to the first alert or the second event (e.g., operations for adjusting one or more settings for the computer system, operations for accepting or rejection a recommended course of action, operations for dismissing the indication without changing settings of the computer system, or other contextual operations based on the content and/or triggering event corresponding to the second notification). In some embodiments, the computer system displays respective indications of respective alerts corresponding to respective events of the second type, and a respective indication of an alert corresponding to an event of the second type includes one or more options for performing operations corresponding to the alert. In some embodiments, different indications for different alerts corresponding to events of the second type include different options for performing operations corresponding to their respective alerts and/or events (e.g., the available options for performing operations depend on the content of the alert, or a characteristic of the triggering event). In some embodiments, the computer system detects a second user input directed to a respective selectable option (e.g., an option for dismissing the alert, an option for displaying additional information or content associated with the alert or event, or joining a real-time communication session associated with the alert or event) of the one or more selectable options, and in response to detecting the second user input, the computer system performs a respective operation (e.g., dismisses the alert, displays additional information or content, or joins the real-time communication session) that corresponds to the first alert or the second event. For example, as described with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 14000 in some circumstances has a different appearance as described in the methods 9000-13000, and 15000-16000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-13000, and 15000-16000). For brevity, these details are not repeated here.
Displaying a respective notification with a first visual appearance based at least in part on a first set of values for a first visual characteristic of a representation of the physical environment at the respective location at which the respective notification is displayed, in accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a first set of values for a first visual characteristic, and displaying the respective notification with a second visual appearance, different from the first visual appearance, that is based at least in part on the second set of values for the first visual characteristic of the representation of the physical environment at the respective location, in accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a second set of values, different from the first set of values, for the first visual characteristic, automatically displays the respective notification with the appropriate visual appearance without further user input (e.g., the user does not need to perform additional user inputs each time the viewpoint of the user changes), and also provides improved visual feedback regarding the user’s real and/or virtual surroundings (e.g., by accurately reflecting the physical environment at the respective location at which the respective notification is displayed).
While a respective view of a three-dimensional environment (e.g., an AR, VR, or MR environment that includes one or more physical objects and/or one or more virtual objects arranged in a three-dimensional arrangement) that includes one or more physical objects (e.g., the representation 7014′ in
In some embodiments, while displaying the respective notifications, the computer system detects a first change in a current viewpoint of a user (e.g., based on movement of at least a portion of the computer system and/or a shift in a virtual viewpoint of the user of the computer system, based on movement of the user’s head or as a whole in the physical environment (e.g., while the user wears the display generation component on his head or over his eyes), and/or based on a request for locomotion that causes the viewpoint of the user to move (e.g., translate, pan, tilt, and rotate) inside the virtual three-dimensional environment) from a first viewpoint associated with the first view of the three-dimensional environment to a second viewpoint associated with a second view of the three-dimensional environment. In response to detecting the first change in the current viewpoint of the user, the computer system updates the respective view of the three-dimensional environment in accordance with the first change of the viewpoint of the user (e.g., from the first view to a second view corresponding to the second viewpoint, optionally, through a sequence of intermediate views corresponding to a sequence of intermediate viewpoints along the movement path of the current viewpoint); and the computer system updates a respective visual appearances of the respective notification in accordance with a change in the first visual characteristic of the representation of the physical environment at the respective location at which the respective notification is displayed (e.g., the respective location maintains its relationship with the field of view provided by the display generation component (e.g., the respective location at which the respective notification is displayed is optionally stationary relative to the currently displayed view, or the respective location at which the respective notification is displayed exhibits a lazy follow behavior relative to the currently displayed view)). In some embodiments, in response to detecting the change in the current viewpoint of the user, in accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a third set of values for the first visual characteristic (e.g., due to the visual characteristics of a third set of objects (e.g., virtual and/or physical objects) and/or lighting (e.g., virtual and/or physical lighting) that can be seen in the line of sight of the user at the current viewpoint of the user), the computer system displays the respective notification with a third visual appearance (e.g., an appearance characterized by a third set of colors, brightness, saturation, tones, and/or spatial distributions of the above) that is based at least in part on the third set of values for the first visual characteristic (e.g., third colors, brightness, saturation, tone, and/or spatial distributions of the above) of the representation of the physical environment at the respective location; and in accordance with a determination that the representation of the physical environment at the respective location at which the respective notification is displayed has a fourth set of values, different from the first set of values and the third set of values, for the first visual characteristic (e.g., due to the visual characteristics of a fourth set of objects (e.g., virtual and/or physical objects) and/or lighting (e.g., virtual and/or physical lighting) that can be seen in the line of sight of the user at the current viewpoint of the user, and/or due to changes in the appearances of the third set of objects that can be seen in the line of sight of the user at the current viewpoint of the user), the computer system displays the respective notification with a fourth visual appearance (e.g., an appearance characterized by a fourth set of colors, brightness, saturation, tones, and/or spatial distributions of the above), different from the first visual appearance and the third visual appearance, that is based at least in part on the fourth set of values for the first visual characteristic (e.g., second colors, brightness, saturation, tone, and/or spatial distributions of the above) of the representation of the physical environment at the respective location. For example, in
In some embodiments, the first visual characteristic is color; displaying the respective notification with the first visual appearance includes displaying the respective notification with a first set of background colors that is based at least in part on a first set of color values of the representation of the physical environment at the respective location; and displaying the respective notification with the second visual appearance includes displaying the respective notification with a second set of background colors, different from the first set of background colors, that is based at least in part on a second set of color values of the representation of the physical environment at the respective location. For example, in some embodiments, the respective notification representation is displayed on a partially translucent platter as the background of the notification content, where the partially translucent platter that takes on color(s) based on color(s) of the physical environment that is behind the respective notification. The partially translucent platter optionally reflects some visual aspects of the physical environment that is behind the respective notification (e.g., an approximate shape of an object behind the respective notification, or a brightness/darkness of an object and/or the physical environment behind the respective notification), in accordance with some embodiments. For example, as described above with reference to
In some embodiments, the respective view of the three-dimensional environment includes one or more virtual objects (e.g., other user interface objects (e.g., user interfaces, controls, windows, and/or other user interface objects), virtual overlays, virtual lighting, virtual décor, or other virtual objects in the virtual or augmented reality environment); the respective location at which the respective notification is displayed is between a first location at which a respective virtual object is displayed and the viewpoint of the user; and the respective notification is displayed with a respective visual appearance that is based at least in part on a respective set of values for the first visual characteristic of the respective virtual object at the first location. For example, in some embodiments, if virtual objects are displayed (or the user transitions to a virtual environment from a physical environment or AR environment), the notification representations that are displayed on partially translucent platters can take on color(s) based on the color(s) of the virtual elements that are behind the notification representations. For example, as described above with reference to
In some embodiments, the respective notification is one of a first group of notifications that correspond to a first group of detected events, and wherein the respective notification is displayed as a representative notification for the first group of notifications. In some embodiments, a representative notification is a notification that is the newest notification in a group of notifications that are received for the same type of event and/or the same application. In some embodiments, the representative notification is displayed with its notification content, and other notifications in the group are displayed as one or more indications behind or adjacent to the representative notification. In some embodiments, the group of notifications are stacked or are not all visible at the same time, and the representative notification is among a subset of the group of notifications that are currently visible. In some embodiments, in response to detecting the first input, the computer system displays a plurality of notifications corresponding to previously detected events at respective locations in the three-dimensional environments, and some (e.g., some but not all) notifications of the plurality of notifications are representative notifications that correspond to respective groups of detected events, while other notifications of the plurality of notifications each correspond to a single detected event that does not fall into the same group as other notifications. For example, in
In some embodiments, the first group of detected events corresponding to a first application (e.g., the first group of detected events are events generated by the first application, and does not include events generated by the computer system or other applications). The computer system displays, in the first view of the three-dimensional environment, a respective representative notification for a second group of notifications that correspond to a second group of events, wherein the second group of detected events corresponds to a second application different from the first application. For example, in some embodiments, if there are multiple notifications for detected events corresponding to a respective application (e.g., a first application, a second application, a third application, and so on), the multiple notifications are displayed as a group (e.g., a first group, a second group, a third group, and so on), with a subset (e.g., one, two, or another number less than all) of the multiple notifications displayed as representative notification(s) for the group of notifications for the respective application. In other words, the notifications are grouped by application, and the entire stack of the notifications for a respective group take on an appearance that is based at least in part on the visual characteristics of the environment behind the stack. For example, as described with reference to
In some embodiments, while displaying the respective notification at the respective location in the three-dimensional environment, the computer system detects a second input that is directed to the respective notification (e.g., a gaze and dwell input, a gaze detected in conjunction with an air pinch gesture, a gaze detected in conjunction with an air pinch and drag gesture, an air pinch and hold gesture, a pinch and drag gesture, a tap gesture, and other types of input directed to the respective notification). In response to detecting the second input that is directed to the respective notification, the computer system performs a respective operation with respect to the respective notification (e.g., displaying more of the first group of notifications, clear the first group of notifications, scroll the notifications displayed concurrently with the respective notification, scrolling the first group of notifications that have been expanded, and other operations related to the respective notification or the group of notifications represented by the notification, or the notifications concurrently displayed with the respective notification). For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to navigate through content to which the second input is directed (e.g., the second input is an air pinch and drag gesture detected when a gaze input is detected in the region occupied by notifications including the respective notification, or on the respective notification, or on the system function menu 7024 described with reference to
In some embodiments, while navigating through of the plurality of notifications, the computer system changes respective visual appearances of the plurality of notifications (e.g., the plurality of notifications take on different sets of values for the first visual characteristic) based at least in part on a respective set of values for the first visual characteristic of respective portions of the three-dimensional environment that are behind the plurality of notifications relative to a current viewpoint of the user, as the plurality of notifications move through different portions of the three-dimensional environment. In some embodiments, the computer system displays an individual notification of the plurality of notifications with a respective visual appearance as described above with reference to the respective notification (e.g., changes in accordance with changes in the appearance of the physical objects/lighting, virtual objects/lighting, changes in accordance with relative movement between the notification and the viewpoint, and in accordance with relative movement between the notification and the three-dimensional environment). In some embodiments, the computer system continually updates the visual appearances of individual notifications in the plurality of notification while scrolling through the plurality of notifications. For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to dismiss content to which the second input is directed (e.g., the second input is a wave of a hand of the user in the air, an air pinch gesture detected when a gaze input is detected on a “close” or “dismiss” affordance displayed adjacent to the respective notification or a plurality of notifications including the respective notification, or the user’s attention being directed to other regions of the environment outside of the region occupied by the notifications, or another type of scroll input), ceasing to display the respective notification (e.g., and any other notifications that are concurrently displayed with the respective notification). For example, as described with reference to
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to expand content to which the second input is directed (e.g., the second input is a pinch and hold gesture, a gaze and dwell input, an air tap gesture detected while the gaze input is detected on the respective notification, or another type of expansion input), displaying an expanded version of the respective notification, including additional content (e.g., additional text content not initially displayed, media content, contact information for a sender of a received message or communication, and/or attachments (e.g., associated with an email notification)) associated with the respective notification that was not included in the respective notification before the second input was detected (e.g., the respective notification includes preview content that is a subset of all available notification content for the respective notification, and the additional content associated with the respective notification includes some notification content that was not included in the preview content), wherein displaying the expanded version of the respective notification includes: in accordance with a determination that the respective notification is a first notification displayed at a first location in the respective view of the three-dimensional environment, displaying first additional content associated with the first notification (e.g., first notification content that was not included in the preview of content included in the respective notification) at the first location. In one example, the respective notification is a notification of a first message from a first sender, and includes an application icon for the messaging application, a portion of the textual content of the first message, the name of the sender, and the timestamp of the first message; and the expanded version of the respective notification includes additional content such as the entirety of the textual content of the first notification, a preview of an attachment (e.g., an image, an animated model, or other graphical or media objects), and one or more controls for performing respective operations associated with the first message (e.g., “reply”, “forward”, “playback”, “save” affordances). In another example, the respective notification is a notification of an online shopping application, and includes the application icon of the online shopping application, a status update for a purchase (“e.g., item shipped”, “item returned”, or “delivery expected at 3pm today”); and the expanded version of the respective notification includes description and images of the items in the order, tracking number of the delivery, and one or more coupons for future purchases. Displaying the expanded version of the respective notification includes: in accordance with a determination that the respective notification is a second notification displayed at a second location different from the first notification displayed at the first location in the respective view of the three-dimensional environment, displaying second additional content (e.g., second notification content that was not included in the preview of content included in the respective notification), different from the first additional content, associated with the second notification at the second location. In one example, the respective notification is a notification of a first email from a first sender, and includes an application icon for the email application, a subject heading of the first email, the name of the sender, and the timestamp of the first email; and the expanded version of the respective notification includes additional content such as an excerpt of the textual content of the first email, a preview of an attachment (e.g., a document, an image, an animated model, or other graphical or media objects), and one or more controls for performing respective operations associated with the first email (e.g., “view in application”, “reply”, “forward” affordances). In another example, the respective notification is a notification of a newsfeed application, and includes the application icon of the newsfeed and one or more headlines for one or more subscribed newsfeed; and the expanded version of the respective notification includes the one or more headlines with excerpts of the corresponding news, one or more thumbnail images corresponding to the one or more headlines, and one or more recommended news headlines and one or more controls for unsubscribe, find similar, or save the newsfeed. Other examples of notifications and their corresponding standard or expanded versions are possible, in accordance with various embodiments. For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to expand content to which the second input is directed (e.g., the second input is a pinch and hold gesture, a gaze and dwell input, an air tap gesture detected while the gaze input is detected on the respective notification, or another type of expansion input) and that the respective notification is a representative notification for a first group of notifications corresponding to a first type of detected events, displaying, concurrently with the respective notification, one or more additional notifications in the first group of notifications that were not displayed at a time when the second input was detected (e.g., if the respective notification is a representative notification for a stack of grouped notifications, the second input causes the stack to be expanded showing individual notifications in the stack); and In one example, the respective notification is a first notification for a first message from a first sender, and the first notification is a latest message for a group of messages from the first sender, and shown as the representative notification for the group of notifications that correspond to the messages received from the first sender; and in response to detecting the second input, the computer system displays respective notifications (e.g., the standard, unexpanded versions of the notifications) for the earlier messages from the first sender that were not initially displayed with the first notification for the first message. In another example, the respective notification is a first notification of a group of notifications for an online shopping application. The group of notifications correspond to a first purchase order made in the online shopping application and the first notification is a lasted notification among the group of notifications, and serves as the representative notification of the group of notifications. In response to detecting the second input, the computer system displays one or more earlier notifications in the group of notifications for the first purchase order, e.g., including notifications for “order confirmed,” “item shipped,” “delivery expected at 3pm today”, and “item delivered.” In another example, the respective notification is a notification of a first email from a first sender, and is a representative notification for a group of notifications for other emails received in the email application. In response to detecting the second input, the computer system displays the respective notifications for other emails received earlier in the email application, where the respective notifications include respective subject headings of the emails, the respective names of the senders, and the respective timestamps of the emails. In another example, the respective notification is a first notification of a newsfeed application, and is a representative notification for a group of notifications corresponding to the same news source, or the same category of news. In response to detecting the second input, the computer system displays multiple notifications in the group of notifications that correspond to other news headlines from the same news source or category of news. Other examples of notifications and their corresponding groups of notifications are possible, in accordance with various embodiments. The respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to expand content to which the second input is directed (e.g., the second input is a pinch and hold gesture, a gaze and dwell input, an air tap gesture detected while the gaze input is detected on the respective notification, or another type of expansion input) and that the respective notification is an individual notification not part of a respective group of notifications, displaying an expanded version of the respective notification (e.g., replacing the standard, unexpanded version of the respective notification), including additional content (e.g., additional text content not initially displayed, media content, contact information for a sender of a received message or communication, and/or attachments (e.g., associated with an email notification)) associated with the respective notification that was not included in the respective notification at the time when the second input was detected. In one example, the respective notification is a notification of a first message from a first sender, and includes an application icon for the messaging application, a portion of the textual content of the first message, the name of the sender, and the timestamp of the first message; and the expanded version of the respective notification includes additional content such as the entirety of the textual content of the first notification, a preview of an attachment (e.g., an image, an animated model, or other graphical or media objects), and one or more controls for performing respective operations associated with the first message (e.g., “reply”, “forward”, “playback”, “save” affordances). In another example, the respective notification is a notification of an online shopping application, and includes the application icon of the online shopping application, a status update for a purchase (“e.g., item shipped”, “item returned”, or “delivery expected at 3pm today”); and the expanded version of the respective notification includes description and images of the items in the order, tracking number of the delivery, and one or more coupons for future purchases. In another example, the respective notification is a notification of a first email from a first sender, and includes an application icon for the email application, a subject heading of the first email, the name of the sender, and the timestamp of the first email; and the expanded version of the respective notification includes additional content such as an excerpt of the textual content of the first email, a preview of an attachment (e.g., a document, an image, an animated model, or other graphical or media objects), and one or more controls for performing respective operations associated with the first email (e.g., “view in application”, “reply”, “forward” affordances). In another example, the respective notification is a notification of a newsfeed application, and includes the application icon of the newsfeed and one or more headlines for one or more subscribed newsfeed; and the expanded version of the respective notification includes the one or more headlines with excerpts of the corresponding news, one or more thumbnail images corresponding to the one or more headlines, and one or more recommended news headlines and one or more controls for unsubscribe, find similar, or save the newsfeed. Other examples of notifications and their corresponding standard or expanded versions are possible, in accordance with various embodiments. For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to expand content to which the second input is directed (e.g., the second input is a pinch and hold gesture, a gaze and dwell input, an air tap gesture detected while the gaze input is detected on the respective notification, or another type of expansion input) and that the respective notification is a representative notification for a first group of notifications corresponding to a first type of detected events: displaying, concurrently with the respective notification, one or more additional notifications in the first group of notifications that were not displayed at a time when the second input was detected (e.g., if the respective notification is a representative notification for a stack of grouped notifications, the second input causes the stack to be expanded showing individual notifications in the stack); and reducing visual prominence (e.g., darkening, making more translucent, reducing in size, and/or ceasing to display completely) of other notifications that were concurrently displayed with the respective notification at the time when the second input was detected (e.g., such that the notifications in the same group as the respective notification are more visually prominent in the environment). For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to dismiss content to which the second input is directed (e.g., the second input is a wave of a hand of the user in the air, an air pinch gesture detected when a gaze input is detected on a “close” or “dismiss” affordance displayed adjacent to the respective notification or a plurality of notifications including the respective notification, or the user’s attention being directed to other regions of the environment outside of the region occupied by the notifications, or another type of scroll input): reducing visual prominence (e.g., darkening, making more translucent, reducing in size, and/or ceasing to display completely) of the respective notification in the respective view of the three-dimensional environment; in accordance with a determination that the second input is directed to a first region relative to the respective notification (e.g., the first region is the location of a dismiss affordance displayed on or adjacent to the respective notification, or the first region is a top corner of the respective notification, or another portion of the respective notification) and that the respective notification is a representative notification for a first group of notifications corresponding to detected events of a first event type, increasing visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of a second notification, different from the respective notification, from the first group of notifications in the respective view of the three-dimensional environment (e.g., replacing display of the respective notification with display of a next notification in the first group of notification, or moving the respective notification to a less prominent position and moving the next notification to the more prominent position previously occupied by the respective notification); and in accordance with a determination that the second input is directed to a second region relative to the respective notification that is different from the first region (e.g., the second region is other portions of the respective notification outside of the first region, or the second region is another closing affordance displayed outside of the respective notification), and that the respective notification is the representative notification for the first group of notifications corresponding to detected events of the first event type, forgoing increasing the visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of another notification from the first group of notifications in the respective view of the three-dimensional environment, after ceasing displaying the respective notification (e.g., the whole first group of notifications is dismissed from the respective view). For example, as described with reference to
In some embodiments, while displaying the second notification at the respective location in the three-dimensional environment after reducing visual prominence of the respective notification in response to the second input, the computer system detects a third input that is directed to the second notification (e.g., a gaze and dwell input, a gaze detected in conjunction with an air pinch gesture, a gaze detected in conjunction with an air pinch and drag gesture, an air pinch and hold gesture, a pinch and drag gesture, a tap gesture, and other types of input directed to the second notification). In response to detecting the third input that is directed to the second notification, and in accordance with a determination that the third input corresponds to a request to dismiss content to which the third input is directed (e.g., the third input is a wave of a hand of the user in the air, an air pinch gesture detected when a gaze input is detected on a “close” or “dismiss” affordance displayed adjacent to the second notification or a plurality of notifications including the second notification, or another type of scroll input): the computer system reduces visual prominence (e.g., darkening, making more translucent, reducing in size, and/or ceasing to display completely) of the second notification in the respective view of the three-dimensional environment; in accordance with a determination that the third input is directed to the first region relative to the second notification (e.g., the first region is the location of a dismiss affordance displayed on or adjacent to the second notification, or the first region is a top corner of the second notification, or another portion of the second notification), the computer system increases visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of a third notification, different from the respective notification and the second notification, from the first group of notifications in the respective view of the three-dimensional environment (e.g., replacing display of the second notification with display of a next notification in the first group of notification, or moving the second notification to a less prominent position and moving the next notification to the more prominent position previously occupied by the second notification); and in accordance with a determination that the third input is directed to the second region relative to the second notification, the computer system forgoes increasing the visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of another notification from the first group of notifications in the respective view of the three-dimensional environment, after ceasing displaying the second notification (e.g., the whole first group of notifications is dismissed from the respective view). For example, as described with reference to
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to dismiss content to which the second input is directed (e.g., the second input is a wave of a hand of the user in the air, an air pinch gesture detected when a gaze input is detected on a “close” or “dismiss” affordance displayed adjacent to the respective notification or a plurality of notifications including the respective notification, or the user’s attention being directed to other regions of the environment outside of the region occupied by the notifications, or another type of scroll input): reducing visual prominence (e.g., darkening, making more translucent, reducing in size, and/or ceasing to display completely) of the respective notification in the respective view of the three-dimensional environment; in accordance with a determination that the second input is directed to a first region relative to the respective notification (e.g., the first region is the location of a dismiss affordance displayed on or adjacent to the respective notification, or the first region is a top corner of the respective notification, or another portion of the respective notification) and that the respective notification is a representative notification for a first group of notifications corresponding to detected events of a first event type, increasing visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of a third notification, different from the respective notification, from the first group of notifications in the respective view of the three-dimensional environment (e.g., replacing display of the respective notification with display of a next notification in the first group of notification, or moving the respective notification to a less prominent position and moving the next notification to the more prominent position previously occupied by the respective notification); and in accordance with a determination that the second input is directed to the first region relative to the respective notification (e.g., the first region is the location of a dismiss affordance displayed on or adjacent to the respective notification, or the first region is a top corner of the respective notification, or another portion of the respective notification) and that the respective notification is an individual notification not part of a group of notifications, forgoing increasing the visual prominence (e.g., increasing visibility by increasing brightness, saturation, and opacity, increasing in size, and/or displaying if not previously visible) of another notification in the respective view of the three-dimensional environment, after ceasing displaying the respective notification (e.g., the respective notification is dismissed from the respective view, and other notifications remain displayed in the respective view). For example, in
In some embodiments, performing, in response to detecting the second input that is directed to the respective notification, the respective operation with respect to the respective notification includes: in accordance with a determination that the second input corresponds to a request to dismiss content to which the second input is directed (e.g., the second input is a wave of a hand of the user in the air, an air pinch gesture detected when a gaze input is detected on a “close” or “dismiss” affordance displayed adjacent to the respective notification or a plurality of notifications including the respective notification, or the user’s attention being directed to other regions of the environment outside of the region occupied by the notifications, or another type of scroll input): in accordance with a determination that the second input is directed to the first region relative to the respective notification (e.g., the first region is the location of a dismiss affordance displayed on or adjacent to the respective notification, or the first region is a top corner of the respective notification, or another portion of the respective notification) and that the respective notification is an individual notification not part of a group of notifications, displaying a fourth notification at the respective location that was occupied by the respective notification at a time when the second input was detected. In some embodiments, the respective notification is replaced by another notification corresponding to a detected event that was not previously displayed. In some embodiments, the respective notification is replaced by another notification of the plurality of notifications that was concurrently visible with the respective notification at the time that the second input was detected. For example, before replacing display of the respective notification with the notification different from the respective notification, a plurality of notifications are displayed in a particular order (e.g., reverse chronological order), and the respective notification is replaced by a notification of the plurality of notifications that follows the respective notification in the particular order (e.g., is the next most recent notification), and the other displayed notifications adjust their positions to maintain the particular order (e.g., each remaining notification of the plurality of notifications that are older than the respective notification take the previous position of the notification immediately before the remaining notification to maintain the particular order), and optionally, a new notification that is not a notification of the plurality of notifications is displayed (e.g., in the previous position of the oldest notification of the plurality of notifications). For example, in
In some embodiments, the respective notification is concurrently displayed with one or more other notifications that correspond to respective detected events (e.g., the respective notification and the one or more other notifications include representative notifications for one or more groups of notifications and/or individual notifications that do not belong to any group of notifications). While concurrently displaying the respective notification and the one or more other notifications that correspond to respective detected events, the computer system detects a third change in a current viewpoint of a user (e.g., based on movement of at least a portion of the computer system and/or a shift in a virtual viewpoint of the user of the computer system, based on movement of the user’s head or as a whole in the physical environment (e.g., while the user wears the display generation component on his head or over his eyes), and/or based on a request for locomotion that causes the viewpoint of the user to move (e.g., translate, pan, tilt, and rotate) inside the virtual three-dimensional environment) from a first viewpoint associated with the first view of the three-dimensional environment to a third viewpoint associated with a third view of the three-dimensional environment. In response to detecting the third change in the current viewpoint of the user: the computer system updates the respective view of the three-dimensional environment in accordance with the third change of the current viewpoint of the user (e.g., from the first view to the third view corresponding to the third viewpoint, optionally, through a sequence of intermediate views corresponding to a sequence of intermediate viewpoints along the movement path of the current viewpoint); and the computer system changes respective visual appearances of the respective notification and the one or more other notifications (e.g., the plurality of notifications take on different sets of values for the first visual characteristic) based at least in part on a respective set of values for the first visual characteristic of respective portions of the three-dimensional environment that are behind the respective notification and the one or more other notifications relative to the third viewpoint of the user (e.g., as the respective portions change due to the movement of the viewpoint). In some embodiments, the computer system displays an individual notification of the plurality of notifications with a respective visual appearance as described above with reference to the respective notification (e.g., changes in accordance with changes in the appearance of the physical objects/lighting, virtual objects/lighting, changes in accordance with relative movement between the notification and the viewpoint, and in accordance with relative movement between the notification and the three-dimensional environment). In some embodiments, the computer system continually updates the visual appearances of individual notifications in the plurality of notification while the viewpoint of the user moves in the three-dimensional environment relative to the respective notification and the one or more other notifications. For example, in
In some embodiments, the computer system maintains respective positions of the respective notification and the one or more other notifications in the three-dimensional environment while the current viewpoint of the user changes from the first viewpoint to the third viewpoint (e.g., the positions of the notifications are locked to the three-dimensional environment, and do not change with the movement of the viewpoint that corresponds to the currently displayed view of the three-dimensional environment). For example, as described with reference to
In some embodiments, while concurrently displaying the respective notification and the one or more other notifications in the respective view of the three-dimensional environment, the computer system detects a fourth input (e.g., an air gesture (e.g., an air tap or an air pinch) detected in conjunction with a gaze input; an air gesture, a touch gesture, an input provided via a controller, a voice command, and/or other types of input) directed to at least one of the respective notification and the one or more other notifications. In response to detecting the fourth input, in accordance with a determination that the fourth input corresponds to a request to move content to which the fourth input is directed, the computer system constrains movement of the respective notification and the one or more other notifications that are concurrently displayed in the respective view of the three-dimensional environment, despite of the fourth input. In some embodiments, constraining the movement of the respective notification and the one or more other notifications include preventing movement of the respective notification and the one or more notifications completely, and ignoring the fourth input. In some embodiments, constraining the movement of the respective notification and the one or more other notifications include providing some visual feedback to the fourth input, such as moving at least one of the respective notification and the one or more notifications in accordance with the fourth input, but limit a range of the movement and/or restoring the moved notification(s) back to their original locations after the fourth input ends. For example, as described with reference to
In some embodiments, prior to detecting the first input, the computer system displays a system user interface (e.g., the system function menu 7024 described in
In some embodiments, the system user interface includes one or more recent notifications that correspond to respective detected events that satisfy timing criteria (e.g., detected events that were detected within a threshold amount of time (e.g., 5 seconds, 10 seconds, 30 seconds, 1 minute, 15 minutes, 30, minutes, 1 hour, 2 hours, or 5 hours)). For example, as described with reference to
In some embodiments, the one or more recent notifications included in the system user interface are scrollable in a first direction (and are optionally not scrollable in a second direction) relative to the respective view of the three-dimensional environment, and the respective notification is concurrently displayed with one or more other notifications in the respective view of the three-dimensional environment, and the respective notification and the one or more other notifications are scrollable in a second direction, different from the first direction, relative to the respective view of the three-dimensional environment. For example, as described with reference to
In some embodiments, the second input that is directed to the respective notification includes a gaze input that is directed to the respective notification (e.g., gaze that satisfies location and/or duration requirements, and/or in conjunction with an air gesture (e.g., an air tap or an air pinch), or other types of gestures or other types of inputs). For example, in
In some embodiments, the second input that is directed to the respective notification includes an air gesture (e.g., an air tap or an air pinch). In some embodiments, the second input includes an air gesture (e.g., an air tap or an air pinch) that is detected while the computer system detects that the attention of the user is directed to the respective notification. In some embodiments, the second input is an air gesture that does not require any gaze component (e.g., the user can perform the second input, and the computer system will display the one or more notifications, regardless of where the attention of the user is directed). In some embodiments, the first input is an air gesture (e.g., an air tap or an air pinch) that is directly detected at the location of the respective notification. For example, as described above with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 15000 in some circumstances has a different appearance as described in the methods 9000-14000, and 16000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-14000, and 16000). For brevity, these details are not repeated here.
The method (16000) relates to hiding and redisplaying different types of user interface elements. For example, one type of user interface element is continuously displayed for the user at a position within the three-dimensional environment based on the user’s current viewpoint of the three-dimensional environment, while another type of user interface element is visually deemphasized as the user moves around in the physical environment, thus changing the user’s viewpoint, and is redisplayed for the user after the user’s viewpoint has settled at a new view. Automatically deemphasizing and/or hiding, and redisplaying user interface objects at updated displayed locations based on the type of user interface object and based on whether the user’s current viewpoint is settled at a position or is still moving, provides real-time visual feedback as the user shifts their viewpoint to different portions of the three-dimensional environment and reduces a number of inputs needed to reinvoke a user interface object that follows the user’s viewpoint. Providing improved visual feedback to the user and reducing the number of inputs needed to perform an operation enhances the operability of the system and makes the user-system interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the system) which, additionally, reduces power usage and improves battery life of the system by enabling the user to use the system more quickly and efficiently.
The computer system displays (16002), via the first display generation component, a first user interface object (e.g., a system user interface object such as a window, one or more system indicators (e.g., indicator 7010 of system function menu described with reference to
While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects (16004), via the one or more input devices, first movement of a current viewpoint of the user from the first viewpoint to a second viewpoint of the user (e.g., change in viewpoint caused by the movement of the user’s head, user’s position as a whole, the display generation component, and/or the one or more cameras that captures the representation of the physical environment in the three-dimensional environment). For example, as described with reference to
In response to detecting the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint (16006), the computer system replaces (16008) display of the first view of the three-dimensional environment with display of a second view of the three-dimensional environment that corresponds to the second viewpoint of the user. For example, as described with reference to
In accordance with a determination that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet first criteria (e.g., does not include a threshold amount of movement in a respective direction, the current viewpoint does not remain substantially stationary at the second viewpoint, and/or does not move the original position of the first user interface object outside of the current view of the three-dimensional environment) and that the first user interface object is of a first type (e.g., first category of objects, objects meeting a first set of conditions, objects having a first set of appearances, behaviors and/or functions), the computer system displays (16010) the first user interface object in the second view of the three-dimensional environment, wherein the first user interface object is displayed at a second position, different from the first position, in the three-dimensional environment and has the first spatial relationship with the second viewpoint of the user. For example, as described with reference to
In accordance with a determination that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet the first criteria (e.g., does not include a threshold amount of movement in a respective direction, the current viewpoint does not remain substantially stationary at the second viewpoint, and/or does not move the original position of the first user interface object outside of the current view of the three-dimensional environment) and that the first user interface object is of a second type (e.g., second category of objects, objects meeting a second set of conditions, objects having a second set of appearances, behaviors and/or functions) different from the first type, the computer system forgoes (16012) displaying the first user interface object at the second position in the three-dimensional environment in the second view of the three-dimensional environment (e.g., the first user interface object no longer has the first spatial relationship with the current viewpoint of the user). For example, as described with reference to
The computer system detects (16014), via the one or more input devices, second movement of the current viewpoint of the user from the second viewpoint to a third viewpoint of the user (e.g., a continuation of the first movement of the current viewpoint or subsequent movement of the viewpoint and/or a movement that starts while displaying the second view of the three-dimensional environment that corresponds to the second viewpoint of the user) (e.g., a change in viewpoint caused by the continuing movement of the user’s head, user’s position as a whole, the display generation component, and/or the one or more cameras that captures the representation of the physical environment in the three-dimensional environment, after the first movement of the current viewpoint is completed). In some embodiments, the continuation of the first movement is movement that occurs within a threshold amount of time after the completion of the first movement of the current viewpoint. In some embodiments, the continuation of the first movement is movement that occurs without a substantial amount of time in which less that a threshold amount of movement has been made (e.g., without a substantial amount of rest or stationary period) after the first movement of the viewpoint. For example, as described with reference to
In response to detecting the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint (16016), the computer system replaces (16018) display of the second view of the three-dimensional environment with display of a third view of the three-dimensional environment that corresponds to the third viewpoint of the user. In accordance with a determination that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria (e.g., includes the threshold amount of movement in a respective direction, the current viewpoint remains substantially stationary at the third viewpoint, and/or the original position of the first user interface object is outside of the current view of the three-dimensional environment), the computer system displays (16020) the first user interface object in the third view of the three-dimensional environment, wherein the first user interface object is displayed at a third position, different from the first position and the second position, in the three-dimensional environment and has the first spatial relationship with the third viewpoint of the user, irrespective of whether the first user interface object is of the first type or the second type. For example, as described with reference to
In some embodiments, user interface objects of the first type include a user interface element that is accessible via a respective input that includes one or more controls for controlling one or more system functions (e.g., as described with reference to system function menu 7024 and/or indicator 7010, and method 8000). For example, if the first user interface object is a user interface element that, when invoked (e.g., by a user input), displays one or more controls for controlling one or more system functions, the first user interface object is of the first type and continuously follows movement of the user’s viewpoint such that the first user interface object remains at a same position relative to the currently displayed view of the environment, and that the first user interface object has a predictable location in the field of view of the user whenever the user wishes to locate it despite of having moved/looked around in the environment. For example, as described with reference to
In some embodiments, user interface objects of the first type include closed captions for concurrently played media content (e.g., audio file, movie, speech input, speech output, voice over, and/or instructions output in enhanced accessibility mode). For example, if the first user interface object includes closed captions for media content that is currently played in the three-dimensional environment, the first user interface object is of the first type and continuously follows movement of the user’s viewpoint such that the first user interface object remains at a same position relative to the currently displayed view of the environment and that the first user interface object has a predictable location in the field of view of the user whenever the user wishes to locate it despite of having moved/looked around in the environment. For example, as described with reference to
In some embodiments, user interface objects of the first type include notifications (e.g., a notification associated with an application of the computer system, optionally, not including notifications and alerts regarding overall system status that are generated by the operating system). For example, if the first user interface object is a notification generated by an application, the first user interface object is of the first type and continuously follows movement of the user’s viewpoint such that the user interface object(s) remain at a same position relative to the currently displayed view of the environment and that the first user interface object has a predictable location in the field of view of the user whenever the user wishes to locate it despite of having moved/looked around in the environment. For example, as described with reference to
In some embodiments, prior to detecting the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint, the computer system displays a second user interface object, distinct from the first user interface object, in the first view of the three-dimensional environment, wherein the second user interface object is of the first type. For example, in some scenarios, two user interface objects of the first type are concurrently displayed in the first view of the three-dimensional environment. In some embodiments, the two user interface objects are different categories of objects of the first type. For example, the first user interface object is a notification while the second user interface object includes closed captions for concurrently played media content. For example, as described with reference to
In some embodiments, moving the first user interface object in the first manner in accordance with the first movement of the current viewpoint of the user includes moving the first user interface object with a first damping factor relative to the first movement of the current viewpoint of the user (e.g., the first user interface object follows the current viewpoint of the user at a first rate and/or with a first modifier (e.g., a first multiplier, and/or a first function of the movement of the viewpoint), relative to the movement of the viewpoint of the user). In some embodiments, the first damping factor defines the manner in which the first user interface object tracks the movement of the current viewpoint of the user. For example, a larger damping factor reduces and/or delays the movement of the first user interface object relative to the movement of the viewpoint of the user by a larger amount, as compared to a smaller damping factor. In some embodiments, different user interface objects are assigned different damping factors and may follow the viewpoint with different delays, rates, and/or simulated inertia. In some embodiments, moving the second user interface object in the second manner in accordance with the first movement of the current viewpoint of the user includes moving the second user interface object with a second damping factor, distinct from the first damping factor, relative to the first movement of the current viewpoint of the user (e.g., the second user interface object follows the current viewpoint of the user at a second rate and/or with a second modifier (e.g., a second multiplier, and/or a second function of the movement of the viewpoint), distinct from the first rate and/or the first modifier, relative to the movement of the viewpoint of the user). For example, as described with reference to
In some embodiments, the user interface objects of the second type include a system setup user interface (e.g., a settings user interface that includes one or more options for selecting the system settings and/or preferences). For example, in some embodiments, if the first user interface object is a settings user interface, the first user interface object is of the second type and does not follow movement of the user’s viewpoint unless a threshold amount of movement of the viewpoint is detected. For example, as described with reference to
In some embodiments, user interface objects of the second type include a system alert (e.g., a notification associated with the operating system and/or the computer system (e.g., not an application of the computer system), an alert regarding a change in system status of the computer system, an alert that requires an explicit user input or attention in order to be dismissed). For example, in some embodiments, if the first user interface object is a system alert, the first user interface object is of the second type and does not follow movement of the user’s viewpoint unless a threshold amount of movement of the viewpoint is detected. For example, as described with reference to
In some embodiments, user interface objects of the second type include a media player for displaying currently played media content (e.g., audio file, movie, video, games, and/or other content for media consumption). In some embodiments, the media player is a miniature user interface window (e.g., having a size that is smaller than one or more application windows) that follows the viewpoint of the user. In some embodiments, the media player is a user interface of a display mode for media content that is optionally smaller than a user interface of a primary display mode, such as a full screen display mode, or a window display mode. In some embodiments, the media player including the media content is displayed concurrently with other active windows of the same application and/or other applications, and continues to playback the media content while the user interacts with the user interface objects of the same and/or other applications and/or the three-dimensional environment. In some embodiments, if the first user interface object is a media player with media playing back in it, the first user interface object is of the second type and does not follow movement of the user’s viewpoint unless a threshold amount of movement of the viewpoint is detected. For example, as described with reference to
In some embodiments, prior to detecting the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint, the computer system displays a third user interface object, distinct from the first user interface object, in the first view of the three-dimensional environment, wherein the third user interface object is of the second type. For example, in some scenarios, one or more user interface objects of the first type are concurrently displayed with one or more user interface objects of the second type in the first view of the three-dimensional environment. In some embodiments, two user interface objects of the second type are different categories of objects of the second type. For example, the first user interface object is a system alert while the third user interface object includes a media player (e.g., a mini media player window, or a reduced version of the regular media player interface) or a settings user interface of the computer system. In some embodiments, in response to detecting the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint: in accordance with a determination that the first user interface object is of a first category of the second type, and that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet the first criteria with respect to the first category of the second type, the computer system forgoes displaying the first user interface object at the second position in the three-dimensional environment in the second view of the three-dimensional environment. In some embodiments, in accordance with a determination that the third user interface object is of a second category, different from the first category, of the second type, and that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet the first criteria with respect to the second category of the second type, the computer system forgoes displaying the third user interface object at a new position in the three-dimensional environment in the second view of the three-dimensional environment (e.g., the first user interface object and the third user interface object no longer have the same respective spatial relationships with the current viewpoint of the user). In some embodiments, the first user interface object and/or the third user interface object remain in the field of view provided via the first display generation component, and/or remain at their respective positions in the three-dimensional environment. In some embodiments, the first user interface object and/or the third user interface object move out of the field of view provided via the first display generation component when their respective positions in the three-dimensional environment are no longer visible in the current field of view of the display generation component. In some embodiments, when the movement of the viewpoint of the user does not meet the first criteria with respect to the respective sub-categories of the second type for the first user interface object and/or the third user interface object, the first and/or third user object are world locked (e.g., remains substantially stationary relative to the three-dimensional environment, including the representations of physical objects and/or virtual objects in the three-dimensional environment that are substantially stationary). In some embodiments, in response to detecting the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint: in accordance with a determination that the first user interface object is of the first category of the second type, that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint meets the first criteria with respect to the first category of the second type (e.g., includes a first threshold amount of movement in a respective direction, the current viewpoint does not remains substantially stationary at the third viewpoint, and/or the original position of the first user interface object is not outside of the current view of the three-dimensional environment), and that the first movement of the current viewpoint of the user from the first viewpoint to the second viewpoint does not meet the first criteria with respect to the second category of the second type (e.g., does not include a second threshold amount of movement in a respective direction, the current viewpoint does not remain substantially stationary at the third viewpoint (e.g., if this is not a requirement for the first category of the second type), and/or the original position of the first user interface object is not outside of the current view of the three-dimensional environment (e.g., if this is not a requirement for the first category of the second type)), the computer system displays the first user interface object in the third view of the three-dimensional environment at the third position that has the first spatial relationship with the third viewpoint of the user without displaying the third user interface object in the third view of the three-dimensional environment at a new position that has a same spatial relationship with the third viewpoint as a previous position of the third user interface object has with the first viewpoint before the first and second movement of the current viewpoint of the user. In some embodiments, when the first criteria are met by the continued movement of the current viewpoint for the first category of object of the second type, the user interface objects of the first category of the second type cease to remain at their original positions in the three-dimensional environment, and is displayed at respective new positions in the updated field of view to have the original spatial relationships with the current viewpoint of the user. However, if the first criteria are not met by the same movement of the current viewpoint for the second category of object of the second type, the user interface objects of the second category of the second type continue to remain at their original positions in the three-dimensional environment until the movement of the viewpoint meet the first criteria with respect to the second category of the second type. In some embodiments, the first criteria are different from the first category and the second category of objects of the second type. In some embodiments, objects of the second type are displayed with delayed follow behavior (e.g., wherein objects of the second type are updated to move and/or follow the current viewpoint of the user after the user has moved by a threshold amount and/or settled at a particular viewpoint). In some embodiments, there are a plurality of categories of user interface objects of the second type wherein different categories of the user interface objects of the second type have different delayed follow behaviors (e.g., a first category is faded out and/or is animated as moving, a second category ceases to be displayed before being redisplayed in the settled view, or otherwise displayed with different follow behavior based on the category of the object)). For example, as described with reference to
In some embodiments, in response to detecting the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint: in according with a determination that the first user interface object is of a first category of the second type, and that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria with respect to the first category of the second type, the computer system displays a first type of delayed follow behavior for the first user interface object before displaying the first user interface object at the third position in the three-dimensional environment; and in according with a determination that the first user interface object is of a second category, different from the first category, of the second type, and that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria with respect to the second category of the second type, the computer system displays a second type of delayed follow behavior, different from the first type of delayed follow behavior, for the first user interface object before displaying the first user interface object at the third position in the three-dimensional environment. For example, as described with reference to
In some embodiments, displaying the first type of delayed follow behavior for the first user interface object before displaying the first user interface object at the third position in the three-dimensional environment includes moving the first user interface object in a third manner (e.g., displayed with a first angle and/or with a first distance of movement up, down, left, right, forward and/or backward) to the third position in the third view of the three-dimensional environment; and displaying the second type of delayed follow behavior, different from the first type of delayed follow behavior, for the first user interface object before displaying the first user interface object at the third position in the three-dimensional environment includes moving the first user interface object in a fourth manner, distinct from the third manner, to the third position in the third view of the three-dimensional environment (e.g., displayed with a second angle and/or with a second distance of movement up, down, left, right, forward and/or backward). In some embodiments, if multiple user interface objects of the second type (e.g., multiple user interface objects of different categories of the second type, and/or with different starting spatial relationships to the current viewpoint before the movement of the viewpoint is detected) are concurrently displayed in the first view of the three-dimensional environment, the first criteria are met at different times during the movement of the viewpoint of the user (e.g., when the viewpoint of the user reaches different positions relative to the three-dimensional environment), and the multiple user interface objects execute different movements (e.g., with different movement angles, directions, speeds, and/or distances to catch up to their respective positions that have the same spatial relationship to the current viewpoint as before the movement of the viewpoint was detected) when the first criteria are met for the different user interface objects. For example, alert 7198-7 is displayed as moving (e.g., from its previous position in the three dimensional environment illustrated in
In some embodiments, the determination that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria with respect to the first category of the second type includes a determination that the third viewpoint has remained as the current viewpoint of the user for at least a first threshold amount of time; and the determination that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria with respect to the second category of the second type includes a determination that the third viewpoint has remained as the current viewpoint of the user for at least a first threshold amount of time, different from the first threshold amount of time. For example, in some embodiments, the current viewpoint of the user is required to have remained substantially unchanged for a respective amount of time (e.g., “settled” at a respective position for a threshold amount of time) before user interface objects of the second type start to follow and catch up to the movement of the viewpoint of the user (e.g., the user interface objects of the second type delay following the viewpoint of the user until the user has settled at a respective viewpoint). In some embodiments, the amount of settle time is different for different categories of the second type of object. For example, in some embodiments, when there are user interface objects belonging to different categories of the second type with different amount of required settlement time, after the current viewpoint of the user reaches the third viewpoint for a first amount of time, the first criteria are met for a first subset of the user interface objects that is of the first category of the second type, and as a result the first subset of user interface objects (and not other user interface objects of categories of the second type) are displayed at respective new positions to reestablish their respective spatial relationships with the third viewpoint; and at a later time, after the current point of the user has reached the third viewpoint for a second amount of time, the first criteria are met for a second subset of the user interface objects that is of the second category of the second type, and as a result, the second subset of user interface objects are displayed at respective new positions to reestablish their respective spatial relationships with the third viewpoint. For example, as described with reference to
In some embodiments, prior to displaying the first user interface object at the third position in the third view of the three-dimensional environment in response to detecting the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint: in accordance with a determination that the first user interface object is of the second type (and, optionally, in accordance with a determination that the first criteria have not been met by the movement of the viewpoint up to this point in time), the computer system changes one or more visual properties of the first user interface object (e.g., fading out, making more translucent, darkening, dimming, ceasing to display, reducing in size, and/or otherwise making it less visually prominent than before) to visually deemphasize the first user interface object in a currently displayed view of the three-dimensional environment during at least a portion of the first movement and the second movement of the current viewpoint of the user; and in accordance with a determination that the first user interface object is of the first type (and, optionally, in accordance with a determination that the first criteria have not been met by the movement of the viewpoint up to this point in time), the computer system changes the one or more visual properties of the first user interface object in the currently displayed view of the three-dimensional environment during at least a portion of the first movement and the second movement of the current viewpoint of the user by less than the one or more visual properties of the first user interface object would be changed to visually deemphasize the first user interface object during at least a portion of the first movement and the second movement of the current viewpoint of the user if the object were an object of the second type (e.g., changing the one or more visual properties by a smaller amount and/or forgoing changing the one or more visual properties). In some embodiments, the first user interface object of the second type follows the viewpoint of the user with a delay and optionally fades out, blurs, ceases to be displayed, or is otherwise visually deemphasized for a period of time before the first criteria are met, e.g., as the viewpoint of the user moves and before the viewpoint settles. In contrast, in some embodiments, the first user interface object of the second type follows the viewpoint of the user with a delay and optionally fades out, blurs, ceases to be displayed, or is otherwise visually deemphasized for a period of time before the first criteria are met, e.g., as the viewpoint of the user moves and before the viewpoint settles. For example, in
In some embodiments, changing the one or more visual properties of the first user interface object (e.g., fading out, making more translucent, darkening, dimming, ceasing to display, reducing in size, and/or otherwise making it less visually prominent than before) to visually deemphasize the first user interface object in the currently displayed view of the three-dimensional environment during at least a portion of the first movement and the second movement of the current viewpoint of the user, includes: while detecting the at least a portion of the first movement and the second movement of the current viewpoint of the user, updating (e.g., gradually increasing, increasing with a changing rate, and/or otherwise dynamically adjusting, optionally, in accordance with elapsed time, and/or movement distance and/or speed of the viewpoint) an amount of change in the one or more visual properties of the first user interface object relative to an original appearance of the first user interface object (e.g., the values of the one or more visual properties of the first user interface object at a time before the movement of the current viewpoint is detected), over a period of time (e.g., a finite amount of time that is visually noticeable by an average user, e.g., more than 1 second, 2 seconds, 5 seconds, or some other threshold amount of time) during the at least a portion of the first movement and the second movement of the current viewpoint of the user (e.g., during the second movement only, during the first movement only, or during a portion of the first movement and a portion of the second movement). For example, as described with reference to
In some embodiments, the determination that the second movement of the current viewpoint of the user from the second viewpoint to the third viewpoint meets the first criteria includes a determination the current viewpoint of the user has less than a threshold amount of movement for at least a respective threshold amount of time (e.g., the viewpoint has remained substantially stationary, while the current viewpoint is the third viewpoint) (e.g., the respective amount of time is a first threshold amount of time for a first category of the second type, a second threshold amount of time for a second category of the second type, or a same threshold amount of time for different categories of the second type). In some embodiments, the first user interface object remains at its original position before it is displayed at the third position in the third view of the three-dimensional environment after the current viewpoint has settled at the position of the third viewpoint for the threshold amount of time with less than the threshold amount movement. In some embodiments, the first user interface object ceases to be displayed or is displayed with an altered appearance of less visual prominence before it is redisplayed at the third position in the third view of the three-dimensional environment after the current viewpoint has settled at the position of the third viewpoint for the threshold amount of time with less than the threshold amount movement. For example, as described with reference to
In some embodiments, in response to detecting at least a portion of the first movement and the second movement of the current viewpoint of the user (e.g., movement from the first viewpoint to the second viewpoint, to an intermediate viewpoint between the first viewpoint and the second viewpoint, or to another intermediate viewpoint between the second viewpoint and the third viewpoint, movement from the first viewpoint to the third viewpoint, or movement from an intermediate viewpoint between the second and third viewpoints to the third viewpoint), in accordance with a determination that the at least a portion of the first movement and the second movement of the current viewpoint of the user meet second criteria (e.g., different from the first criteria) (e.g., second criteria include criteria based on distance, user’s attention, spatial relationship between the first user interface object and the viewpoint, and/or spatial relationship between the first user interface object and the currently displayed view, and/or related timing or duration of one or more the above) and that the first user interface object is of a third type, different from the first type and the second type, the computer system ceases display of the first user interface object. In some embodiments, in response to detecting at least a portion of the first movement and the second movement of the current viewpoint of the user, in accordance with a determination that the at least a portion of the first movement and the second movement of the current viewpoint of the user does not meet the second criteria, and that the first user interface object is of the third type, maintaining display of the first user interface object. In some embodiments, after ceasing display of the first user interface object: in response to detecting the second movement of the current viewpoint of the user: in accordance with a determination that the second movement of the current viewpoint meets the first criteria and that the first user interface object is of the third type, forgoes displaying the first user interface object in the third view of the three-dimensional environment. In some embodiments, after ceasing display of the first user interface object, in response to detecting the second movement of the current viewpoint of the user, in accordance with a determination that the second movement of the current viewpoint of the user does not meet the first criteria, and that the first user interface object is of the third type, forgoing displaying the first user interface object in the third view (e.g., after the first user interface object has ceased to be displayed, it is not redisplayed in response to the second movement that meets the first criteria). For example, in some embodiments, user interface objects of the third type do not follow the viewpoint of the user. In some embodiments, user interface objects of the first type continuously follow the viewpoint of the user, user interface objects of the second type follow the viewpoint of the user with a delay (e.g., after the viewpoint is settled, or after a threshold amount of movement of the viewpoint has been detected, and/or other conditions related to the movement of the viewpoint, time, and spatial relationships between the object, the field of view, and/or the viewpoint), and objects of the third type ceases to be displayed (e.g., is closed or dismissed) after the movement of the viewpoint is started, and is optionally displayed in response to a subsequent user input that corresponds to an explicit request to redisplay the user interface object at a new position. For example, as described with reference to
In some embodiments, while displaying the third view of the three-dimensional environment, the computer system detects a user input that corresponds to a request to display a user interface object of the third type; and in response to detecting the user input that corresponds to a request to display a user interface object of the third type, in accordance with a determination that the first user interface object is of the third type and that the first user interface object has ceased to be displayed during the at least a portion of the first movement and the second movement of the current viewpoint of the user, redisplays the first user interface object in the third view of the three-dimensional environment (e.g., optionally at a same position relative to the current viewpoint of the user in the third view of the three-dimensional environment). In some embodiments, in response to detecting the user input that corresponds to the request to display a user interface object of the third type, in accordance with a determination that the first user interface object is not of the third type, the computer system forgoes displaying the first user interface object in the third view of the three-dimensional environment. In some embodiments, in response to detecting the user input that corresponds to the request to display a user interface object of the third type, in accordance with a determination that the first user interface object is of the third type, and that the first user interface object is currently displayed in the third view of the three-dimensional environment, the computer system maintains display of the first user interface object in the third view of the three-dimensional environment. In some embodiments, in response to detecting a user input that does not correspond to a request to display a user interface object of the third type, in accordance with a determination that the first user interface object is of the third type and that the first user interface object has ceased to be displayed during the at least a portion of the first movement and the second movement of the current viewpoint of the user, the computer system forgoes redisplaying the first user interface object in the third view of the three-dimensional environment. For example, as described with reference to
In some embodiments, user interface objects of the third type includes one or more instances of a control center user interface (e.g., that enables the user to control one or more settings and/or one or more quick-access functions), a notification user interface that displays one or more previously received notifications (e.g., a user interface element that displays notifications that have been saved, if any), an expanded version of a notification (e.g., the system optionally expands a notification from a reduced or normal version to an expanded version that includes additional notification content of the notification and/or additional controls in response to a user gaze input directed to the reduced or normal version of the notification), a home user interface (e.g., that displays one or more application icons or other system functions), a keyboard, and/or a virtual assistant, respectively. For example, as described with reference to
In some embodiments, in response to detecting the at least a portion of the first movement and the second movement of the current viewpoint of the user (e.g., movement from the first viewpoint to the second viewpoint, to an intermediate viewpoint between the first viewpoint and the second viewpoint, or to another intermediate viewpoint between the second viewpoint and the third viewpoint, movement from the first viewpoint to the third viewpoint, or movement from an intermediate viewpoint between the second and third viewpoints to the third viewpoint): in accordance with a determination that the at least a portion of the first movement and the second movement of the current viewpoint of the user meets third criteria, the computer system ceases display of the first user interface object, wherein: the second criteria include a requirement that a viewpoint of the user move outside of a second region (e.g., a safe zone in which movement of the viewpoint of the user will not cause the first user interface object to disappear) of the three-dimensional environment in order for the third criteria to be met; the third criteria include a requirement that a viewpoint of the user move outside of a third region (e.g., a safe zone in which movement of the viewpoint of the user will not cause the first user interface object to disappear) of the three-dimensional environment in order for the third criteria to be met; and the third region has a different size and/or shape than the second region. In some embodiments, in response to detecting the at least a portion of the first movement and the second movement of the current viewpoint of the user, in accordance with a determination that the at least a portion of the first movement and the second movement of the current viewpoint does not meet the third criteria, the computer system maintains display of the first user interface object. For example, in some embodiments, objects of different types and optionally different categories (e.g., the second type, different categories of the second type, the third type, and/or different categories of the third type) have different safe zones (e.g., a safe viewing zone to view a respective user interface object) in which movement of the viewpoint do not cause the objects to cease to be displayed. In other words, while the user’s viewpoint is in the safe zone for a respective object of the second type or the third type, the user interface object continues to be displayed; and if the user’s viewpoint is outside of the safe zone for a respective object, the user interface object ceases to be displayed (optionally, before it is redisplayed when the first criteria are met, or when it is redisplayed in response to an explicit user input, depending on whether the object is an object of the second type, or an object of the third type). In some embodiments, the third type of user interface object has a safe zone that includes the user interface object, such that the user is enabled to approach (e.g., get close to) the user interface object and optionally interact with the user interface object through direct manipulation using a gesture at the location of the user interface object. For example, while the user and/or the user’s viewpoint is within the safe zone, the first user interface object is maintained in the display at a same position within the three dimensional environment (e.g., without regard to the movement of a current viewpoint of the user), and while the user and/or the user’s viewpoint is outside of the safe zone, the first user interface object is redisplayed in front of the user (e.g., with the same spatial relationship relative to the view of the three-dimensional environment corresponding to the current viewpoint of the user). In some embodiments, the safe zone is defined relative to the position of the first user interface object (e.g., as a U-shape with the first user interface object located in the bottom portion of the U-shape, a rectangular box with the first user interface object located at the center of the rectangular box, or other irregular shapes with the first user interface object located inside or outside). For example, as described with reference to
In some embodiments, the third region is larger than the second region (e.g., user interface objects of the third type have a larger safe zone than the user interface objects of the second type). For example, as described with reference to
In some embodiments, the third region has a different shape from the second region. (e.g., irregular shapes (e.g., a U-shape, a shape other than a circle, square, or rectangle) that are based on the shape and/or nature of the user interface object, and optionally, based on the posture of the user and how the user is viewing the three-dimensional environment (e.g., reclined, sitting, or standing)). For example, as described with reference to
In some embodiments, the third region extends to a location that is close enough to the location of the first user interface object in the three-dimensional environment (e.g., the third region extends to, is near, or encompasses the first user interface object) to enable the user to provide inputs (e.g., direct inputs) to a location in the three-dimensional environment at which the first user interface object is located. For example, as described with reference to
In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, and 16000 may be interchanged, substituted, and/or added between these methods. For example, the first user interface object (e.g., system control indicator) in the method 16000 in some circumstances has a different appearance as described in the methods 9000-15000, and the user interface elements that are displayed (e.g., the plurality of affordances for accessing system functions of the first computer system) may be replaced by, or concurrently displayed with, other user interface elements (e.g., additional content associated with a notification, a user interface that includes an affordance for joining a communication session, and other user interface elements in the methods 9000-15000). For brevity, these details are not repeated here.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best use the invention and various described embodiments with various modifications as are suited to the particular use contemplated.
As described above, one aspect of the present technology is the gathering and use of data available from various sources to improve XR experiences of users. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, twitter IDs, home addresses, data or records relating to a user’s health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.
The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to improve an XR experience of a user. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used to provide insights into a user’s general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.
The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. Such policies should be easily accessible by users, and should be updated as the collection and/or use of data changes. Personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection/sharing should occur after receiving the informed consent of the users. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted for the particular types of personal information data being collected and/or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations. For instance, in the US, collection of or access to certain health data may be governed by federal and/or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly. Hence different privacy practices should be maintained for different personal data types in each country.
Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of XR experiences, the present technology can be configured to allow users to select to “opt in” or “opt out” of participation in the collection of personal information data during registration for services or anytime thereafter. In another example, users can select not to provide data for customization of services. In yet another example, users can select to limit the length of time data is maintained or entirely prohibit the development of a customized service. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.
Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user’s privacy. De-identification may be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and/or other methods.
Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, an XR experience can generated by inferring preferences based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the service, or publicly available information.
This application claims priority to U.S. Provisional Application Serial No. 63/445,698, filed Feb. 14, 2023, U.S. Provisional Application Serial No. 63/409,594, filed Sep. 23, 2022, and U.S. Provisional Application Serial No. 63/310,970, filed Feb. 16, 2022, each of which are incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63445698 | Feb 2023 | US | |
| 63409594 | Sep 2022 | US | |
| 63310970 | Feb 2022 | US |
