Patent Application
20250157147
This application claims the benefit of priority to U.S. Provisional Patent Application No. 63/597,677, entitled, “Atlased Coverage Mesh For Three-Dimensional Rendering”, filed on Nov. 9, 2023, the disclosure of which is hereby incorporated herein in its entirety.
The present description relates generally to electronic devices including, for example, atlased coverage meshes for three-dimensional rendering.
Electronic devices often include displays on which scenes are displayed. Typically, the scenes include objects that are arranged in two-dimensions on a two-dimensional display of an electronic device. Some electronic devices have two-dimensional displays with the capability of displaying scenes that appear to be arranged in three dimensions.
Certain features of the subject technology are set forth in the appended claims. However, for purpose of explanation, several implementations of the subject technology are set forth in the following figures.
The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and can be practiced using one or more other implementations. In one or more implementations, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.
A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).
There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.
Electronic devices often display information that is distributed, in two dimensions, on a two-dimensional display, such as a liquid crystal display or light-emitting diode (e.g., an organic light-emitting diode, or OLED), on which each display pixel is arranged to be viewed concurrently by both eyes of a user. Some electronic devices, such as XR headsets, have the capability of displaying information on one or more two-dimensional displays that appears, to a viewer of the display, to be distributed in three-dimensions (e.g., by displaying arranging the display(s) and/or portions thereof to be viewed differently by the two eyes of a user).
Implementations of the subject technology described herein can provide for efficient rendering of three-dimensional scenes, such as XR scenes. The efficient rendering may be used to render a view of a three-dimensional scene that is initially represented (e.g., stored) using one or more discrete layers of scene content that are spatially distributed across the three-dimensional scene. In some examples, the representation of the three-dimensional scene may be in the form of a multi-plane image (MPI) and/or multi-sphere image (MSI) representation.
The subject disclosure may provide various improvements and/or advantages over existing procedures for rendering scenes from multi-plane image (MPI) and/or multi-sphere image (MSI). In particular, rather than, or in addition to, storing rectilinear cutouts of image objects from an MPI or MSI in an atlas image, coverage meshes for image objects can be generated and stored in a mesh atlas.
In various implementations, the mesh atlas can be generated from an image atlas containing images of objects, or meshes can be generated directly from an MPI, MSI, or other layer-based three-dimensional representation, and subsequently packed together in a mesh atlas. As discussed in further detail hereinafter, the mesh atlas can provide efficiencies, relative to rendering using only an image atlas, by reducing the number of transparent pixels in a scene that are unnecessarily rendered, facilitating use of GPU depth testing to avoid rendering of occluded pixels that are later replaced by opaque foreground pixels, and/or facilitating compact packing of image objects into an atlas. In one or more implementations, a mesh atlas can be generated in advance and used, at rendering time, for efficient rendering of a three-dimensional scene from a current point of view of a user.
The system architecture 100 includes an electronic device 105, an electronic device 110, an electronic device 115, and a server 120. For explanatory purposes, the system architecture 100 is illustrated in
The electronic device 105 may be smartphone, a tablet device, or a wearable device such as a head mountable portable system, that includes a display system capable of presenting a visualization of an extended reality environment to a user 101. The electronic device 105 may be powered with a battery and/or any other power supply. In an example, the display system of the electronic device 105 provides a stereoscopic presentation of the extended reality environment, enabling a three-dimensional visual display of a rendering of a particular scene, to the user. In one or more implementations, instead of, or in addition to, utilizing the electronic device 105 to access an extended reality environment, the user may use an electronic device 104, such as a tablet, watch, mobile device, and the like.
The electronic device 105 may include one or more cameras such as camera(s) 150 (e.g., visible light cameras, infrared cameras, etc.) Further, the electronic device 105 may include various sensors 152 including, but not limited to, cameras, image sensors, touch sensors, microphones, inertial measurement units (IMU), heart rate sensors, temperature sensors, Lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communications sensors, etc.) Moreover, the electronic device 105 may include hardware elements that can receive user input such as hardware buttons or switches. User input detected by such sensors and/or hardware elements correspond to various input modalities for interacting with virtual content displayed within a given extended reality environment. For example, such input modalities may include, but not limited to, facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, head tracking, pose tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breath, temperature, electroencephalogram, olfactory), recognizing speech or audio (e.g., particular hotwords), and activating buttons or switches, etc. The electronic device 105 may also detect and/or classify physical objects in the physical environment of the electronic device 105.
The electronic device 105 may be communicatively coupled to a base device such as the electronic device 110 and/or the electronic device 115. Such a base device may, in general, include more computing resources and/or available power in comparison with the electronic device 105. In an example, the electronic device 105 may operate in various modes. For instance, the electronic device 105 can operate in a standalone mode independent of any base device. When the electronic device 105 operates in the standalone mode, the number of input modalities may be constrained by power limitations of the electronic device 105 such as available battery power of the device. In response to power limitations, the electronic device 105 may deactivate certain sensors within the device itself to preserve battery power.
The electronic device 105 may also operate in a wireless tethered mode (e.g., connected via a wireless connection with a base device), working in conjunction with a given base device. The electronic device 105 may also work in a connected mode where the electronic device 105 is physically connected to a base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device is charging and/or providing power to the electronic device 105 while physically connected).
When the electronic device 105 operates in the wireless tethered mode or the connected mode, a least a portion of processing user inputs and/or rendering the extended reality environment may be offloaded to the base device thereby reducing processing burdens on the electronic device 105. For instance, in an implementation, the electronic device 105 works in conjunction with the electronic device 110 or the electronic device 115 to generate an extended reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the extended reality environment in a real-time manner. In an example, the electronic device 105 provides a rendering of a scene corresponding to the extended reality environment that can be perceived by the user and interacted with in a real-time manner. Additionally, as part of presenting the rendered scene, the electronic device 105 may provide sound, and/or haptic or tactile feedback to the user. The content of a given rendered scene may be dependent on available processing capability, network availability and capacity, available battery power, and current system workload.
The electronic device 105 may also detect events that have occurred within the scene of the extended reality environment. Examples of such events include detecting a presence of a living being such as a person or a pet, a particular person, entity, or object in the scene. Detected physical objects may be classified by electronic device 105, electronic device 110, and/or electronic device 115 and the location, position, size, dimensions, shape, and/or other characteristics of the physical objects can be used to provide physical anchor objects for an XR application generating virtual content, such as a UI of an application, for display within the XR environment.
It is further appreciated that the electronic device 110 and/or the electronic device 115 can also generate such extended reality environments either working in conjunction with the electronic device 105 or independently of the electronic device 105.
The network 106 may communicatively (directly or indirectly) couple, for example, the electronic device 105, the electronic device 110 and/or the electronic device 115 with the server 120 and/or one or more electronic devices of one or more other users. In one or more implementations, the network 106 may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet.
The electronic device 110 may include a touchscreen and may be, for example, a smartphone, a portable computing device such as a laptop computer, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like, any other appropriate device that includes, for example, processing circuitry, memory, a display, and/or communications circuitry for communicating with one or more other devices. In one or more implementations, the electronic device 110 may not include a touchscreen but may support touchscreen-like gestures, such as in an extended reality environment. In one or more implementations, the electronic device 110 may include a touchpad. In
The electronic device 104 may include a touchscreen and may be, for example, a smartphone, a portable computing device such as a laptop computer, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like, any other appropriate device that includes, for example, processing circuitry, memory, a display, and/or communications circuitry for communicating with one or more other devices. In one or more implementations, the electronic device 104 may not include a touchscreen but may support touchscreen-like gestures, such as in an extended reality environment. In one or more implementations, the electronic device 104 may include a touchpad. In
The electronic device 115 may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In
The server 120 may form all or part of a network of computers or a group of servers 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software, and includes specific hardware (e.g., processors, graphics processors and other specialized or custom processors) for rendering and generating content such as graphics, images, video, audio and multi-media files for extended reality environments. In an implementation, the server 120 may function as a cloud storage server that stores any of the aforementioned extended reality content generated by the above-discussed devices and/or the server 120.
In the example of
As shown in
As shown in
In one or more implementations, the information that is used by the electronic device 105 to generate the XR environment of
For example,
In the example of
As illustrated in
For example,
In the example of
In order to render a view of a three-dimensional scene from using the MSI representation 501, a viewpoint 510 may be selected. The viewpoint 510 may be a reference viewpoint, and may include a viewing location 511, and a viewing angle. In the example of
As shown, in the rendered view 512 from the viewpoint 510, the object 502 appears partially in front of the object 506 and appears larger, relative to the size of the object 506, than in the MSI representation 501 due to the shorter distance d1, versus the distance d4 of the location of the object 506. In this example, the object 504 appears to the right of, and separated from, the objects 502 and 506. The rendered view 512 may be displayed by a display of an electronic device, such as the electronic device 105.
Although spherical layers at constant distances are depicted in
For example, the MPI representation 601 of
The viewpoints 510 and 610 of
The MSI representation 501 and the MPI representation 601 are merely illustrative of representations of three-dimensional scenes that include multiple discrete layers of scene content that are spatially distributed across the three-dimensional scene. In various other examples, the discrete layers of a representation of a three-dimensional scene can have shapes other than spherical or planar shapes. Whether in an MSI representation, an MPI representation, or another representation of the three-dimensional scene that includes multiple discrete layers of scene content that are spatially distributed across the scene, each layer may have an associated Red, Green, Blue, α (RGBα) texture, and can be placed at a predetermined (e.g., fixed) depth, distance, and/or radius from a reference viewpoint in the scene. The RGB channels of the texture of each layer may capture the color content of the scene, and the alpha channel of each layer may identify the (e.g., approximate) depths at which content from that layer is located in the scene. For example, objects that are close to the reference viewpoint in the scene may be captured by opaque RGBα content in the textures for layer(s) with smaller depths/distances/radii, and objects at relatively further distances may be represented by opaque RGBα content in the deeper/further/larger layers. By using a finite number of discrete layers, these representations of the three-dimensional scene may discretize the depth information for the scene.
In one or more implementations, relevant information for rendering a view from the MSI representation 501 or the MPI representation 601 can be stored in the form of a two-dimensional image containing representations of the objects therein. For example,
In order to render a view of the three-dimensional scene, each layer of the MSI representation 501 or the MPI representation 601 may be projected to a viewpoint (e.g., viewpoint 510 or 710), and an alpha-composite may be performed from back-to-front of each layer's RGBα texture. For example, performing the alpha-composite may include rendering each of the cutout images 802, 804, and 806 at the distance corresponding to its respective layer in the MSI representation 501 or the MPI representation 601, and according to the RGB values of the cutout images. However, rendering based on the MSI representation 501 or the MPI representation 601, or based on the two-dimensional image 800 generated therefrom, may result in a large number of transparent pixels being rendered without any visible benefit in the rendering. For example, rendering the cutout image 802 would include rendering the pixels corresponding to the object 502 and (e.g., unnecessarily) rendering the transparent pixels 808 surrounding the object 502 within the cutout image 802. Similarly, rendering the cutout image 804 would include rendering the pixels corresponding to the object 504 and (e.g., unnecessarily) rendering the transparent pixels 810 surrounding the object 504 within the cutout image 804. More complex object shapes can result in even more transparent pixels rendered than those shown for objects 502 and 504. Moreover, rendering from back to front, based on the MSI representation 501 or the MPI representation 601, or based on the two-dimensional image 800 generated therefrom, can cause occluded background content (e.g., an occluded portion of object 506 in the examples of
Aspects of the subject technology can address these and/or other performance limitations of rendering MPIs/MSIs, such as by providing atlased alpha coverage meshes. For example, as illustrated by
For example, the coverage meshes 902, 904, and 906 may be generated by identifying the non-transparent pixels in each layer and/or for each object (e.g., using the a values in the cutout images 802, 804, and 806, or in the MSI representation 501 or the MPI representation 601), and identifying a set of vertices 910 for the coverage mesh for each layer and/or each object. Each vertex 910 may include three coordinates in the three-dimensional space in which an object or other scene component is to be rendered to appear, and two coordinates that map to the pixels of the object or other scene component in the two-dimensional image 800. For example, in one or more implementations, in order to generate the vertices 910, a quad can be created per opaque pixel of each object. One or more mesh simplification operations can then be applied to reduce the polygon count and create a coarser coverage mesh for each object. For example, there exists a tradeoff between the coarseness of the coverage meshes and the tightness to which the coverage meshes fit the opaque scene content. There is also a tradeoff between the benefit of reducing the rendering operations for transparent pixels, and increasing the number of vertices to be processed when rendering using the coverage meshes (e.g., using a coarse mesh will reduce the vertex count of the mesh, but will increase the number of transparent pixels needlessly rendered, and vice versa). In one or more implementations, the coarseness of the coverage meshes may be tuned, using a rendering-optimization process, to balance work for processing the number of vertices to the work for rendering the resulting number of excess transparent pixels.
As shown in
Once the mesh atlas 1000 is generated, rendering a view, such as the view 512, the view 712, or any of the views of
Using (e.g., by performing depth testing of the vertices, such as by a graphics processing unit (GPU), during rendering) the vertices of the coverage meshes in the mesh atlas 1000, which indicate the three-dimensional locations of the objects 502, 504, and 506 (e.g., without referring back to the layer of the original representation of the three-dimensional scene), the rendering may include rendering the objects 502, 504, and 506 from front (e.g., locations that are apparently closest to the user 101) to back (e.g., locations that are apparently further from the viewpoint of the user 101). For example, the object 502 may be rendered first, and only portions of the objects 504 and 506 that are not occluded by the object 502 may then be rendered. In this way, the rendering process can identify occluded portions of the objects 502, 504, and/or 506 before rendering those occluded portions, so that the occluded portions are not rendered (e.g., as they would be later overwritten by foreground portions of others of the objects 502, 504, and/or 506), and further efficiency in the rendering of the view of the three-dimensional scene can be achieved.
Referring back to
In the example of
In one or more implementations, generating the mesh atlas directly from the representation of the three-dimensional scene that includes discrete layers of scene content that are spatially distributed across the three-dimensional scene (e.g., directly from the MSI representation 501 or the MPI representation 601), before generating the two-dimensional image 800, can facilitate additional efficiencies, such as by facilitating compact packing of the mesh atlas and the image atlas. For example,
As shown in
In one or more implementations, representing a three-dimensional scene using multiple discrete layers (e.g., an MSI representation, an MPI representation, or other layer-based representation), and/or compacting the information from that representation into a mesh atlas 1000 and a two-dimensional image 800, can be particularly useful for three-dimensional scenes that are static or nearly static. However, in one or more implementations, a mesh atlas 1000 and a two-dimensional image 800 may be generated for rendering of a three-dimensional scene that includes some animated and/or moving content. For example, in one or more implementations, one or more of the objects 502, 504, and 506 may be configured to bounce, wiggle, vibrate, grow, shrink, or otherwise move or change, when a user interacts with that object. In order to accommodate small movements and/or changes of an object (e.g., movements and/or changes over distances that are smaller than a largest dimension of the object), the coverage mesh for the moving object may be expanded. For example,
As shown in
In one or more implementations, for objects with continuous movements across the scene, larger movements than the size of the object, and/or other animations, the mesh atlas 1000 and the two-dimensional image 800 may be generated or updated for each of several time periods during the movement or animation. In this way, the movement and/or animation of the object within each time period may be small, and the expanded coverage mesh may cover the area within which the object moves within that time period.
As illustrated in
In one or more implementations, each of the multiple discrete layers of scene content may represent a respective depth in the three-dimensional scene. In one example, each of the multiple discrete layers may include a spherical layer. In another example, each of the discrete layers may include a planar layer.
At block 1504, for multiple objects (e.g., objects 204, 214, 400, 402, 404, 406, 502, 504, and/or 506) represented in the three-dimensional scene, one or more coverage meshes corresponding to the multiple objects (e.g., coverage meshes 902, 904, and/or 906 for objects 502, 504, and/or 506) may be generated (e.g., by an electronic device, such as electronic device 105) from the representation that includes the multiple discrete layers. For example, at least one of the one or more coverage meshes corresponding to the multiple objects may include vertex information for a boundary (e.g., boundary 960) of one of the multiple objects. For example, the vertex information for a coverage mesh may include three-dimensional coordinates for the locations each of multiple vertices (e.g., vertices 910) in the three-dimensional scene, and two-dimensional coordinates of each of the multiple vertices in the two-dimensional image (e.g., coordinates mapping to the representation of the corresponding object, for that coverage mesh, in the two-dimensional image).
In one or more implementations, the process 1500 may also include determining a number of vertices for the one of the multiple objects by performing an optimization process based on candidate numbers of the vertices and a number of transparent pixels associated with the one of the plurality of objects. For example, the optimization process may include selecting, from the candidate numbers of vertices, a number of vertices that optimally reduces the number of transparent pixels to be rendered, in balance with a cost of processing each vertex. In one or more implementations, generating the at least one of the one or more coverage meshes for the one of the multiple objects may include generating the vertex information for the boundary of the one of the multiple objects by obtaining the vertex information for each of the determined number of vertices for the one of the multiple objects.
In one or more implementations, generating the one or more coverage meshes corresponding to the multiple of objects may include generating, for a first one of the multiple objects (e.g., object 504 or object 506), a single respective coverage mesh (e.g., including only portion(s) 920) corresponding to all pixels of the first one of the plurality of objects, and generating, for a second one of the multiple objects (e.g., object 502), first and second respective coverage meshes. The first respective coverage mesh (e.g., a portion 920) may correspond to only opaque pixels of the second one of the plurality of objects, and the second respective coverage mesh (e.g., a portion 922) may correspond to a set of opaque pixels of the second one of the plurality of objects and to a set of transparent pixels outside a boundary (e.g., boundary 960) of the second one of the plurality of objects.
At block 1506, a mesh atlas (e.g., mesh atlas 1000) may be stored (e.g., by an electronic device, such as electronic device 105). The mesh atlas may include the one or more coverage meshes corresponding to the multiple objects. The mesh atlas may be stored in association with a two-dimensional image (e.g., two-dimensional image 800) containing representations (e.g., cutout images 802, 804, and/or 806) of the multiple objects. In one or more implementations, storing the mesh atlas may include storing the mesh atlas as metadata for the two-dimensional image containing the representations of the plurality of objects. In one or more implementations, the mesh atlas may be stored as a list or table of vertex information, the vertex information including, for each object, a set of coordinates for each of a set of vertices.
In one or more implementations, generating the one or more coverage meshes corresponding to the multiple objects at block 1504 may include generating the two-dimensional image containing the representations of the multiple objects, and generating the one or more coverage meshes corresponding to the multiple objects based on the two-dimensional image (e.g., based on the alpha values in the two-dimensional image). In one or more other implementations, generating the one or more coverage meshes corresponding to the multiple objects may include generating the one or more coverage meshes corresponding to the multiple objects based on the representation of the three-dimensional scene, and the process 1500 may also include generating the two-dimensional image based on the one or more coverage meshes corresponding to the multiple objects and based on the representation of the three-dimensional scene (e.g., by extracting cutout images from the multiple discrete layers according to vertex information in the respective coverage meshes).
At block 1508, the two-dimensional image and the mesh atlas may be provided (e.g., to a renderer at an electronic device, such as electronic device 105) for rendering of the three-dimensional scene (e.g., by an electronic device, such as electronic device 105). The process 1500 may also include rendering a view of the three-dimensional scene using the mesh atlas and the two-dimensional image. Rendering the view of the three-dimensional scene may include performing some or all of the process 1600 of
For example,
As illustrated in
At block 1604, the electronic device may obtain a mesh atlas (e.g., mesh atlas 1000) associated with the two-dimensional image. The mesh atlas and the two-dimensional image may be based on a representation (e.g., MSI representation 501, MPI representation 601, or another representation) of a three-dimensional scene (e.g., the three-dimensional scene of any of
At block 1606, the electronic device may obtain a viewpoint (e.g., viewpoint 510, 610, or 710) for viewing the three-dimensional scene. For example, obtaining the viewpoint may include determining a location of the electronic device and/or a user thereof, determining a pose of the electronic device and/or a user thereof, and/or determining a gaze location or direction of a user of the electronic device. For example, in a use case in which the electronic device is implemented as a head mountable device and the user is wearing the head mountable device, the viewpoint may change as the user moves their head and/or body, and/or looks around the three-dimensional (e.g., XR) scene.
At block 1608, the electronic device may render the three-dimensional scene from the viewpoint using the mesh atlas and the two-dimensional image. For example, the rendering may include identifying, using the mesh atlas, a portion of the two-dimensional image corresponding to an object in the three-dimensional scene; and rendering a portion of the three-dimensional scene by applying the portion of the two-dimensional image to a location in the three-dimensional scene, the location defined by the mesh atlas.
In one or more implementations, rendering the three-dimensional scene may include rendering, based on the mesh atlas, at least a subset of the multiple respective objects in an order from a nearest (e.g., to the electronic device or to a user of the electronic device) one of the subset of the multiple respective objects to a furthest (e.g., from the electronic device or to a user of the electronic device) one of the subset of the multiple respective objects.
In one or more implementations, the mesh atlas may include, for one of the multiple respective objects, a first coverage mesh (e.g., a portion 920) corresponding only to opaque pixels of the one of the plurality of respective objects, and a second coverage mesh (e.g., a portion 922) corresponding to a set of opaque pixels of the one of the multiple respective objects and to a set of transparent pixels outside a boundary (e.g., boundary 960) of the one of the multiple respective objects. In these implementations, the rendering may include rendering the one of the multiple respective objects by: rendering a first portion of the one of the multiple respective objects using depth testing based on the first coverage mesh; and rendering a second portion of the one of the multiple respective objects based on the second coverage mesh and without performing depth testing for the second portion of the one of the plurality of respective objects.
As described above, aspects of the subject technology may include the collection of data. The present disclosure contemplates that in some instances, this collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses data, image data, audio data, environment data, gaze data, image data, location data, pose data, or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.
The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used for providing rendering using atlased coverage meshes from a particular viewpoint. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used, in accordance with the user's preferences to provide insights into their general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.
The present disclosure contemplates that those entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities would be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. Such information regarding the use of personal data should be prominently and easily accessible by users, and should be updated as the collection and/or use of data changes. Personal information from users should be collected for legitimate uses only. Further, such collection/sharing should occur only after receiving the consent of the users or other legitimate basis specified in applicable law. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted for the particular types of personal information data being collected and/or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations which may serve to impose a higher standard. For instance, in the US, collection of or access to certain health data may be governed by federal and/or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly.
Despite the foregoing, the present disclosure also contemplates implementations in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of providing rendering using atlased coverage meshes from a particular viewpoint, the present technology can be configured to allow users to select to “opt in” or “opt out” of participation in the collection of personal information data during registration for services or anytime thereafter. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.
Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user's privacy. De-identification may be facilitated, when appropriate, by removing identifiers, controlling the amount or specificity of data stored (e.g., collecting location data at city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and/or other methods such as differential privacy.
Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data.
The bus 1710 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computing device 1700. In one or more implementations, the bus 1710 communicatively connects the one or more processing unit(s) 1714 with the ROM 1712, the system memory 1704, and the permanent storage device 1702. From these various memory units, the one or more processing unit(s) 1714 retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The one or more processing unit(s) 1714 can be a single processor or a multi-core processor in different implementations.
The ROM 1712 stores static data and instructions that are needed by the one or more processing unit(s) 1714 and other modules of the computing device 1700. The permanent storage device 1702, on the other hand, may be a read-and-write memory device. The permanent storage device 1702 may be a non-volatile memory unit that stores instructions and data even when the computing device 1700 is off. In one or more implementations, a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as the permanent storage device 1702.
In one or more implementations, a removable storage device (such as a flash drive, and its corresponding disk drive) may be used as the permanent storage device 1702. Like the permanent storage device 1702, the system memory 1704 may be a read-and-write memory device. However, unlike the permanent storage device 1702, the system memory 1704 may be a volatile read-and-write memory, such as random access memory. The system memory 1704 may store any of the instructions and data that one or more processing unit(s) 1714 may need at runtime. In one or more implementations, the processes of the subject disclosure are stored in the system memory 1704, the permanent storage device 1702, and/or the ROM 1712. From these various memory units, the one or more processing unit(s) 1714 retrieves instructions to execute and data to process in order to execute the processes of one or more implementations.
The bus 1710 also connects to the input and output device interfaces 1706 and 1708. The input device interface 1706 enables a user to communicate information and select commands to the computing device 1700. Input devices that may be used with the input device interface 1706 may include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output device interface 1708 may enable, for example, the display of images generated by computing device 1700. Output devices that may be used with the output device interface 1708 may include, for example, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.
One or more implementations may include devices that function as both input and output devices, such as a touchscreen. In these implementations, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Finally, as shown in
Implementations within the scope of the present disclosure can be partially or entirely realized using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage medium also can be non-transitory in nature.
The computer-readable storage medium can be any storage medium that can be read, written, or otherwise accessed by a general purpose or special purpose computing device, including any processing electronics and/or processing circuitry capable of executing instructions. For example, without limitation, the computer-readable medium can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. The computer-readable medium also can include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.
Further, the computer-readable storage medium can include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.
Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be realized as executable or non-executable machine code or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Further, instructions also can be realized as or can include data. Computer-executable instructions also can be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those of skill in the art, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without varying the underlying logic, function, processing, and output.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits, such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions that are stored on the circuit itself.
Those of skill in the art would appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods, and algorithms have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. Various components and blocks may be arranged differently (e.g., arranged in a different order, or partitioned in a different way) all without departing from the scope of the subject technology.
It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components (e.g., computer program products) and systems can generally be integrated together in a single software product or packaged into multiple software products.
As used in this specification and any claims of this application, the terms “base station”, “receiver”, “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” means displaying on an electronic device.
As used herein, the phrase “at least one of” preceding a series of items, with the term “and” or “or” to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase “at least one of” does not require selection of at least one of each item listed; rather, the phrase allows a meaning that includes at least one of any one of the items, and/or at least one of any combination of the items, and/or at least one of each of the items. By way of example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and/or at least one of each of A, B, and C.
The predicate words “configured to”, “operable to”, and “programmed to” do not imply any particular tangible or intangible modification of a subject, but, rather, are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or a component may also mean the processor being programmed to monitor and control the operation or the processor being operable to monitor and control the operation. Likewise, a processor configured to execute code can be construed as a processor programmed to execute code or operable to execute code.
Phrases such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment, another embodiment, some implementations, one or more implementations, a configuration, the configuration, another configuration, some configurations, one or more configurations, the subject technology, the disclosure, the present disclosure, other variations thereof and alike are for convenience and do not imply that a disclosure relating to such phrase(s) is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. A disclosure relating to such phrase(s) may apply to all configurations, or one or more configurations. A disclosure relating to such phrase(s) may provide one or more examples. A phrase such as an aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to other foregoing phrases.
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration”. Any embodiment described herein as “exemplary” or as an “example” is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term “include”, “have”, or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term “comprise” as “comprise” is interpreted when employed as a transitional word in a claim.
All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112 (f) unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for”.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more”. Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.
| Number | Date | Country | |
|---|---|---|---|
| 63597677 | Nov 2023 | US |
