Patent Grant
12267545
This invention is related to connected mobile computing systems, methods, and configurations, and more specifically to content provisioning systems, mobile computing systems, methods, and configurations featuring at least one wearable component which may be utilized for virtual and/or augmented reality operation
Content provisioning systems that have one or more augmented reality systems have become popular for viewing the real world with a superimposition of digital content. A content provisioning system may for example include a mobile device such as head-worn viewing component. The content provisioning system may further include a resource device having a resource device data set including content a storage medium. The resource device transmits the content to the mobile device. The mobile device has an output device connected capable of providing an output that can be sensed by a user.
The invention provides a content provisioning system including a mobile device that may have a mobile device processor, a mobile device communication interface connected to the mobile device processor and a first resource device communication interface and under the control of the mobile device processor to receive first content transmitted by the first resource device transmitter, and a mobile device output device connected to the mobile device processor and under control of the mobile device processor capable of providing an output that can be sensed by a user.
The content provisioning system may further include a first resource device that may have a first resource device processor, a first resource device storage medium, and a first resource device data set including first content on the first resource device storage medium, the first resource device communication interface forming part of the first resource device and connected to the first resource device processor and being under the control of the first resource device processor.
The content provisioning system may include that the first resource device is at first location, wherein the mobile device communication interface creates a first connection with the first resource device, and wherein the content is first content specific to a first geographic parameter of the first connection.
The content provisioning system may further include a second resource device that may have a second resource device processor, a second resource device storage medium, a second resource device data set including second content on the second resource device storage medium, and a second resource device communication interface forming part of the second resource device and connected to the second resource device processor and being under the control of the second resource device processor, wherein the second resource device is at second location, wherein the mobile device communication interface creates a second connection with the second resource device, and wherein the content is second content specific to a second geographic parameter of the second connection,
The content provisioning system may include that the mobile device includes head-worn viewing component which is couplable to the head of the user and the first and second content provide the user with at least one of additional content, enhanced content and information that pertains to a particular view of the world as seen by the user.
The content provisioning system may further include a localization island for the user to enter wherein specific features have been pre-configured to be located and interpreted by the mobile device to determine the geographic parameter relative to the world around the user.
The content provisioning system may include that the specific features are visually-detectable features.
The content provisioning system may include that the specific features are wireless connectivity-related features.
The content provisioning system may further include a plurality of sensors connected to the head-worn viewing component that are used by the mobile device to determine the geographic parameter relative to the world around the user.
The content provisioning system may further include a user interface configured to allow the user to at least one of intake, utilize, view, and bypass certain information of the first or second content.
The content provisioning system may include that the connection is a wireless connection.
The content provisioning system may include that the first resource device is at first location, wherein the mobile device has a sensor that detects a first feature at the first location and the first feature is used to determine a first geographic parameter associated with the first feature, and wherein the content is first content specific to a first geographic parameter.
The content provisioning system may include the second resource device is at second location, wherein the mobile device has a sensor that detects a second feature at the second location and the second feature is used to determine a second geographic parameter associated with the second feature, and wherein the first content is updated with second content specific to a second geographic parameter.
The content provisioning system may include that the mobile device includes head-worn viewing component which is couplable to the head of the user and the first and second content provide the user with at least one of additional content, enhanced content and information that pertains to a particular view of the world as seen by the user.
The content provisioning system may further include a spatial computing layer between the mobile device and a resource layer having a plurality of data sources and programmed to receive data resources, integrate the data resources to determine an integrated profile, and determine the first content based on the integrated profile.
The content provisioning system may include that the spatial computing layer may include a spatial computing resource device that may have a spatial computing resource device processor, a spatial computing resource device storage medium, and a spatial computing resource device data set on the spatial computing resource device storage medium and executable by the processor to receive the data resources, integrate the data resources to determine an integrated profile, and determine the first content based on the integrated profile.
The content provisioning system may further include an abstraction and arbitration layer interposed between the mobile device and the resource layer and programmed to make workload decisions, and distribute tasks based on the workload decisions.
The content provisioning system may further include a camera device that takes images of a physical world around the mobile device, wherein the images are used to make the workload decisions.
The content provisioning system may further include a camera device that takes images of a physical world around the mobile device, wherein the images form one of the data resources.
The content provisioning system may include that the first resource device is an edge resource device, wherein the mobile device communication interface includes one or more mobile device receivers connected to the mobile device processor and to a second resource device communication interface in parallel with the connection with the first resource device to receive second content.
The content provisioning system may include that the second resource device is a fog resource device having a second latency that is slower than the first latency.
The content provisioning system may include that the mobile device communication interface includes one or more mobile device receivers connected to the mobile device processor and to a third resource device communication interface in parallel with the connection with the second resource device to receive third content transmitted by the third resource device transmitter, wherein the third resource device is a cloud resource device having a third latency that is slower than the second latency
The content provisioning system may include that the connection to the edge resource device is through a cell tower and the connection to the fog resource device is through a Wi-Fi connection device.
The content provisioning system may include that the cell tower is connected to the fog resource device.
The content provisioning system may include that the Wi-Fi connection device is connected to the fog resource device.
The content provisioning system may further include at least one camera to capture at least first and second images, wherein the mobile device processor transmits the first image to the edge resource device for faster processing and the second image to the fog resource device for slower processing.
The content provisioning system may include that the at least one camera is a room camera that takes the first image of the user.
The content provisioning system may further include a sensor providing a sensor input into a processor, a pose estimator, executable by a processor, to calculate to a pose of the mobile device, including at least one of a location and an orientation of the mobile device, based on the sensor input, a steerable wireless connector that creates a steerable wireless connection between the mobile device and the edge resource device, and a steering system connected to the pose estimator and having an output that provides an input into the steerable wireless connector to steer the steerable wireless connection to at least improve the connection.
The content provisioning system may include that the steerable wireless connector is a phased array antennae.
The content provisioning system may include that the steerable wireless connector is a radar hologram type of transmission connector.
The content provisioning system may further include an arbitrator function executable by a processor to determine how many edge and fog resources are available through the edge and fog resource devices respectively send processing tasks to the edge and fog resources according to the determination of the resources that are available, and receive results back from the edge and fog resources.
The content provisioning system may include that the arbitrator function is executable by the processor to combine the results from the edge and fog resources.
The content provisioning system may further include a runtime controller function executable by the processor to determine whether a process is a runtime process or not, if the determination is made that the task is a runtime process then, executing the task immediately without making the determination with the arbitrator function, and if the determination is made that the task is not a runtime process then making the determination with the arbitrator function.
The content provisioning system may further include a plurality of edge resource devices, data exchanging between a plurality of the edge resource devices and the fog resource device, the data including points in space captured by different sensors and sent to the edge resource devices, and a superpoint calculation function, executable by a processor, to determine superpoints, being select ones of the points where the data from two or more of the edge resource devices overlap.
The content provisioning system may further include a plurality of multiple mobile devices, wherein each superpoint is used in each mobile device for localization, orientation or pose estimation of the respective mobile device.
The content provisioning system may further include a context trigger function, executable with at processor, to generate a context trigger for a group of the superpoints and store the context trigger on a computer-readable medium.
The content provisioning system may further include a rendering engine, executable by the mobile device processor, wherein the context trigger is used as a handle for a rendering of an object based on the first content.
The content provisioning system may further include a rendering function executable by the mobile device processor to connect the mobile device to a plurality of resource devices, wherein each resource device receives a respective rendering request, to receive a rendering from each one of the remote devices based on the respective rendering requests, compare the renderings to determine a preferred rendering, and select, with the mobile device processor, the preferred rendering as the first content transmitted by the first resource device transmitter.
The content provisioning system may include that the renderings form a system having a polynomial prediction for rendering frames into the future where the mobile device is predicted to be posed or looking.
The invention also provides a method of providing content including connecting, under control of a mobile device processor of a mobile device, a mobile device communication interface of the mobile device to a first resource device communication interface of a first resource device, and receiving, with the mobile device communication interface under control of the mobile device processor, first content transmitted by the first resource device transmitter.
The method may further include storing, under control of a first resource device processor, a first resource device data set including first content on a first resource device storage medium connected to the first resource device processor, and transmitting, with a first resource device communication interface connected to the first resource device processor and under the control of the first resource device processor the first content.
The method may include that the first resource device is at first location, wherein the mobile device communication interface creates a first connection with the first resource device, and wherein the content is first content specific to a first geographic parameter of the first connection.
The method may further include storing, under control of a second resource device processor, a second resource device data set including second content on a second resource device storage medium connected to the second resource device processor, and transmitting, with a second resource device communication interface connected to the second resource device processor and under the control of the second resource device processor the second content, wherein the second resource device is at second location, wherein the mobile device communication interface creates a second connection with the second resource device, and wherein the content is second content specific to a second geographic parameter of the second connection.
The method may include that the mobile device includes head-worn viewing component which is couplable to the head of the user and the first and second content provide the user with at least one of additional content, enhanced content and information that pertains to a particular view of the world as seen by the user.
The method may include that the user enters a localization island wherein specific features have been pre-configured to be located and interpreted by the mobile device to determine the geographic parameter relative to the world around the user.
The method may include that the specific features are visually-detectable features.
The method may include that the specific features are wireless connectivity-related features.
The method may include that a plurality of sensors are connected to the head-worn viewing component that are used by the mobile device to determine the geographic parameter relative to the world around the user.
The method may further include receiving through a user interface an input from the user to at least one of intake, utilize, view, and bypass certain information of the first or second content.
The method may include that the connection is a wireless connection.
The method may include that the first resource device is at first location, wherein the mobile device has a sensor that detects a first feature at the first location and the first feature is used to determine a first geographic parameter associated with the first feature, and wherein the content is first content specific to a first geographic parameter.
The method may include that the second resource device is at second location, wherein the mobile device has a sensor that detects a second feature at the second location and the second feature is used to determine a second geographic parameter associated with the second feature, and wherein the first content is updated with second content specific to a second geographic parameter.
The method may include that the mobile device includes head-worn viewing component which is couplable to the head of the user and the first and second content provide the user with at least one of additional content, enhanced content and information that pertains to a particular view of the world as seen by the user.
The method may further include receiving, by a spatial computing layer between the mobile device and a resource layer having a plurality of data sources, data resources, integrating, by the spatial computing layer, the data resources to determine an integrated profile, and determining, by the spatial computing layer, the first content based on the integrated profile.
The method may include that the spatial computing layer may include a spatial computing resource device having a spatial computing resource device processor; a spatial computing resource device storage medium, and a spatial computing resource device data set on the spatial computing resource device storage medium and executable by the processor to receive the data resources, integrate the data resources to determine an integrated profile, and determine the first content based on the integrated profile.
The method may further include making, with an abstraction and arbitration layer interposed between the mobile device and the resource layer, workload decisions, and distributing, with the abstraction and arbitration layer, tasks based on the workload
The method may further include taking, with a camera device, images of a physical world around the mobile device, wherein the images are used to make the workload decisions.
The method may further include taking, with a camera device, images of a physical world around the mobile device, wherein the images form one of the data resources.
The method may include that the first resource device is an edge resource device, further including connecting, under control of a mobile device processor of a mobile device and in parallel with the connection with the first resource device, a mobile device communication interface of the mobile device to a second resource device communication interface of a second resource device, receiving, with the mobile device communication interface under control of the mobile device processor, second content transmitted by the second resource device transmitter.
The method may include that the second resource device is a fog resource device having a second latency that is slower than the first latency.
The method may further include connecting, under control of a mobile device processor of a mobile device and in parallel with the connection with the second resource device, a mobile device communication interface of the mobile device to a third resource device communication interface of a third resource device, wherein the third resource device is a cloud resource device having a third latency that is slower than the second latency, and receiving, with the mobile device communication interface under control of the mobile device processor, third content transmitted by the third resource device transmitter.
The method may include that the connection to the edge resource device is through a cell tower and the connection to the fog resource device is through a Wi-Fi connection device.
The method may include that the cell tower is connected to the fog resource device.
The method may include that the Wi-Fi connection device is connected to the fog resource device.
The method may further include capturing at least first and second images with at least one camera, wherein the mobile device processor transmits the first image to the edge resource device and the second image to the fog resource device.
The method may include that the at least one camera is a room camera that takes the first image of the user.
The method may further include receiving, by a processor, a sensor input, determining, with the processor, a pose of the mobile device, including at least one of a location and an orientation of the mobile device, based on the sensor input, and steering, with the processor, a steerable wireless connector that creates a wireless connection between the mobile device to the edge resource device based on the pose to at least improve the connection.
The method may include that the steerable wireless connector is a phased array antennae.
The method may include that the steerable wireless connector is a radar hologram type of transmission connector.
The method may further include determining, with an arbitrator function executed by a processor, how many edge and fog resources are available through the edge and fog resource devices respectively, sending, with the arbitrator function processing tasks to the edge and fog resources according to the determination of the resources that are available, and receiving, with the arbitrator function, results back from the edge and fog resources.
The method may further include combining, with the arbitrator function, the results from the edge and fog resources.
The method may further include determining, by the mobile device processor, whether a process is a runtime process or not, if the determination is made that the task is a runtime process then, executing the task immediately without making the determination with the arbitrator function, and if the determination is made that the task is not a runtime process then making the determination with the arbitrator function.
The method may further include exchanging data between a plurality of edge resource devices and the fog resource device, the data including points in space captured by different sensors and sent to the edge resource devices, and determining superpoints, being select ones of the points where the data from two or more of the edge resource devices overlap.
The method may further include using each superpoint in multiple mobile devices for localization, orientation or pose estimation of the respective mobile device.
The method may further include generating, with a processor, a context trigger for a group of the superpoints, and storing, with the processor, the context trigger on a computer-readable medium.
The method may further include using the context trigger as a handle for a rendering of an object based on the first content.
The method may further include connecting, under control of the mobile device processor, the mobile device to a plurality of resource devices, transmitting, by the mobile device processor, one or more rendering requests, wherein each resource device receives a respective rendering request, receiving, with the mobile device processor, a rendering from each one of the remote devices based on the respective rendering requests, comparing, with the mobile device processors, the renderings to determine a preferred rendering, and selecting, with the mobile device communication interface under control of the mobile device processor, the preferred rendering first content transmitted by the first resource device transmitter.
The method may include that the renderings form a system having a polynomial prediction for rendering frames into the future where the mobile device is predicted to be posed or looking.
The invention is further described by way of example with reference to the accompanying drawings, wherein:
The content provisioning system of
Referring to
The mobile computing system may be configured such that the user selects certain aspects of his computing experience for the day. For example, through a graphical user interface, voice controls, and/or gestures, the user may input to the mobile computing system that he will have a typical workday, usual route there, stopping at the park for a brief walk on the way home. The mobile computing system has “artificial intelligence” aspects so that it uses integration with an electronic calendar of the user to provisionally understand his schedule, subject to quick confirmations. For example, as he is departing for work, the system may be configured to say or show: “headed to work, usual route and usual computing configuration”, and this usual route may be garnered from previous GPS and/or mobile triangulation data through his mobile computing system. The “usual computing configuration” may be customized by the user and subject to regulations, for example, the system may be configured to only present certain non-occlusive visuals, no advertisements, and no shopping or other information not pertinent to driving while the user is driving, and to provide an audio version of a news program or current favorite audiobook while the user is driving on his way to work. As the user navigates the drive on the way to work, he may leave connectivity with his home wireless device (40) and enter or maintain connectivity with other wireless devices (42, 44, 46, 48). Each of these wireless devices may be configured to provide the user's mobile computing system with information pertinent to the user's experience at relatively low latency (i.e., by storing locally certain information which may be pertinent to the user at that location).
For example, as the user travels from point A (80) to point B (82) to point C (84), a local wireless device (44) around point C (84) may be configured to pass to the user's mobile system geometric information which may be utilized on the user's mobile computing system for highlighting where a trench is being created at such location, so that the user clearly visualizes and/or understands the hazard while driving past, and this geometric information (which may feature a highlighted outline of the trench, for example, may also feature one or more photos or other non-geometric information) may be locally stored on the local wireless device (44) so that it does not need to be pulled from more remote resources which may involve greater latency in getting the information to the driver. In addition to lowering latency, local storage may also function to decrease the overall compute load on the user's mobile computing system, because the mobile system may receive information that it otherwise would have had to generate or build itself based upon sensors, for example, which may comprise part of the locally mobile hardware.
Once the user arrives at the parking lot of his work (24), the system may, for example, be configured to detect walking velocity and be configured by the user to review with the user his schedule for the day, via an integration with his computerized calendaring system, as he is walking up to the office. Certain additional information not resident on his locally mobile computing system may be pulled from local sources (48, 50, for example) which may feature certain storage capacity, to again facilitate smaller mobile overhead and lower latency versus direct cloud connectivity.
Referring to
Referring to
Similarly, as the user navigates a walk (28) through the park (26), shown in magnified view in
Referring to
As described above, to decrease latency and generally increase useful access to pertinent location-based information, wireless devices with localized storage resources, such as those depicted in
The mobile computing system may be customizable by the user to present information filtered on a time-domain basis, such as by how old or “stale” such information is. For example, the user may be able to configure the system to only provide traffic information while he is driving that is 10 minutes old or newer, etc. (i.e., the time domain aspect may be customized/configurable). Alternatively, the user may be able to configure the system to only present architectural (i.e., position of walls within a building) that is 1 year old or newer etc. (i.e., the time domain aspect may be customized/configurable).
Referring to
For example, as shown in the example of
Referring again to
According to additional detail described with reference to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
The next depicted layer is a security/encryption layer (252). This layer is configured to insulate and protect the user from other systems or users which may want to gain access to the user's data or meta data, which may include, for example, what the user likes, what the user does, where the user is located. Technologies such as blockchain may be utilized to assist in securing and configuring such a security layer, and interacting, for example, with a digital identification which may be configured to securely represent who the user is in the digital world, and may be facilitated by biometric authentication techniques.
A next adjacently positioned layer is the human-centered integrated spatial computing layer (254), which also may be referred to under the tradename, “MagicVerse”. This spatial computing layer also is shown in
Referring ahead to
Referring back to
The next layer illustrated in
One portion of the three outermost layers is depicted missing to represent the fact that the real physical world (264) is part of the integrated system, in other words, the world may be utilized to assist in computing, to make various decisions, and to identify various items. The content provisioning system includes a camera device that takes images of a physical world around the mobile device. The images may be used to make the workload decisions. The images may form one of the data resources.
Referring to
In various embodiments, such configurations are controlled to distribute workloads to the various resources depending upon the need for computing workload versus latency. For example, referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring back to
Referring to
Referring to
Referring to
Referring to
As in an MMO-like system, content is stored based upon the coordinate system that the user interacts within. This may be accomplished in a database wherein a specific piece of content may have a JSON (JavaScript object notation) which describes position, orientation, state, time stamp, owner, etc. This stored information is generally is not replicated locally on the user's XR device until the user crosses a location-based boundary and begins download of the content in the predicted future. In order to predict the future, machine learning and/or deep learning algorithms may be configured to use information such as speed, acceleration, distance, direction, behavior, and other factors. This content may be configured to be always located in the same position unless interacted with by one or more users to be re-located, either by manual means (i.e., such as drag and drop using an XR interface) or by programming manipulation. Once new content is downloaded based upon the aforementioned factors, it may be saved onto a local computational device, such as that shown in the form of the computing pack (6) in reference to
Referring to
Referring to
Referring to
Referring to
What may be termed “Spatial Understanding” is an important skill for machines that have to either directly interact with the physical 3D world (e.g., a robot that walks and picks up thrash) or machines that indirectly interact with the 3D world (e.g., Mixed Reality glasses that create high-quality 3D graphics that respect 3D world geometry). Humans generally already have excellent spatial reasoning skills, but even “smart glasses” that are worn by a user must perform their own version of spatial reasoning. Computer scientists have developed certain algorithms suitable for extraction of 3D data and spatial reasoning about the world. The notion of “superpoints” has been described above and in the associated incorporated references, herein we describe a superpoint-based formulation of spatial reasoning that heavily relies on a stream of input images.
As noted above, one particular class of 3D-building algorithms that are pertinent to spatial computing configurations are typically referred to as simultaneous localization and mapping (“SLAM”). SLAM systems may be configured to take as input a sequence of images (color or depth images) and other sensor readings (e.g., inertial measurement units (“IMUs”)) and provide real-time localization of the current device pose. A pose is typically the rotation matrix R and translation vector t corresponding to the camera coordinate system. A SLAM algorithm may be configured to be able to produce pose because behind the scenes it interleaves two core operations: Map-building and Localization against the map.
The term “Visual SLAM” may be used in reference to a variant of SLAM that heavily relies on camera images. As opposed to 3D scanners such as LIDAR which the robotics community developed to help in performing SLAM for industrial applications, cameras are significantly smaller, are much more prevalent, cheaper, and easier to work with. Modern off-the-shelf camera modules are tiny and with camera images it is possible to build a very small and efficient Visual SLAM systems.
What may be termed “Monocular Visual SLAM” is a variant of Visual SLAM that uses a single camera. The major benefit of using a single camera as opposed to two or more is the reduced client form factor. With two cameras, great care must be taken to keep the multi-camera assembly rigid. The two major drawbacks of monocular visual SLAM are the following:
1. Monocular visual SLAM generally cannot recover the 3D structure when there is no parallax, such as the camera system being rotated about the camera center. This is particularly problematic when the system starts out initially with no 3D structure and there is insufficient parallax motion during the initialization of the 3D map inside the algorithm.
2. The second challenge is that monocular visual SLAM generally is unable to recover the absolute scale of the world. For any given monocular trajectory with recovered point depths Zi, it is possible to multiply all point 3D coordinates by a scalar alpha and scale the translation by alpha as well.
However, a small amount of extra information, such as odometry readings from local IMUs or depth data from an RGBD sensor is sufficient to prevent the monocular algorithms from degenerating in general.
A Visual SLAM system operates on images and produces both poses and a 3D map. The entire algorithm may be broken down into two stages: the frontend and the backend. The goal of the frontend is to extract salient 2D image features and describe them so that the raw RGB images are no longer necessary. The task of the frontend is often referred to as “data abstraction” because high-dimensional images are reduced to 2D points and descriptors whose nearest neighbor relationships hold. For properly trained descriptors, we can take the Euclidean distance between them to determine if the correspond to the same physical 3D point—but Euclidean distances on raw images are not meaningful. The goal of the backend is to take the abstractions from the frontend, namely the extracted 2D points and descriptors, and stitch them together to create a 3D map.
As described above and in the incorporated reference documentation, superpoint is a term that may be used for a convolutional deep learning-based frontend, designed for monocular visual SLAM. Traditional computer vision frontends for visual SLAM may consist of hand-crafted 2D keypoint detectors and descriptors. Traditional methods typically follow these steps: 1.) extract 2D key points, 2.) crop patches from the image around the extracted 2D keypoints, 3.) compute descriptors for each patch. “Deep Learning” configurations allow us to train a single multi-headed convolutional neural network that jointly performs the interest point and descriptor computation.
Given a sequence of 2D points tracked in an image, “bundle adjustment” is a term utilized for algorithms which may be used to optimize jointly for the 3D structure and camera poses that best explain the 2D point observations. The algorithms may be configured to minimize the re-projection errors (or rectification error) of the 3D points using a non-linear least squares formulation.
Superpoint, as a deep learning formulation of feature extraction, may be configured as a network that contains very little manual-engineering (i.e., manual input). While the network may be designed to take an image as input and provide both 2d point locations and associated descriptors, it generally cannot do so until it is first trained on a proper dataset. The knowledge of how to extract those 2D points is never explicitly delineated. The network is trained using back propagation on a labeled dataset.
We previously have described how to train the first part of superpoint: the key point localization head. This may be accomplished by creating a large synthetic dataset of corners and the resulting network (which is just like superpoint but does not contain descriptors), this may be referred to as a “MagicPoint”.
Once one has a MagicPoint, one has a way of extracting 2D key points from an arbitrary image. Generally, one still needs to 1.) improve the performance of key point detection on real-world images and 2.) add the descriptor to the network. Improving performance on real-world images means that one must train on real-world images. Adding the descriptor means that one generally must train on pairs of images because one must provide the algorithms with positive and negative pairs of key points to learn their descriptor embeddings.
A so-called MagicPoint configuration may be run on real images using synthetic homographies with a procedure that may be called “Homographic Adaptation”. This provides for better labels on those images than running MagicPoint only once per image. Synthetic homographies may be used to take an input image I and create two warped versions I′ and I″. Since a composition of homographies is still a homography, one may train on the pair of images (I′, I″) with the homography between them.
The resulting system may be called superpoint_v1 and is the result of running MagicPoint on random real-world images using Homographic Adaptation.
At this point one has what may be termed a “superpoint_v1”, a convolutional frontend that provides all that is necessary for a barebones Visual Odometry or Visual SLAM system. However, superpoint_v1 was trained using random (non-temporally ordered images) and all image-image variations were due to synthetic homographies and synthetic noise. In order to make a better superpoint system, one may re-train “superpoint_v2” on real-world sequences using the output of SLAM.
Referring to
The backend may be configured to perform two tasks: providing the pose of the current measurements (Localization, see block two in
The localization module (see block three in
The map update module may be configured to take as input the current map as well as the current image observations with the estimated pose and produces a new updated map. The map may be updated by minimizing the reprojection error of the 3D points in a large number of keyframes. This may be achieved by setting up a bundle adjustment optimization problem as is commonly done in known “Structure-from-Motion” computer vision literature. The bundle adjustment problem is a non-linear least squares optimization problem and can be efficiently solved using the second order Levenberg—Marquardt algorithm.
Generally, a system which uses superpoint may be configured such that it needs to make certain design decisions regarding the layout of the computations. On one extreme, all computations involved in dealing with the camera sensor and forming a well-behaved intensity image must happen locally (see block One in
Local on-device SLAM (100% local). On one extreme, it is possible to take all that is required for localization and mapping and put that directly on the device with the cameras. In this scenario, blocks one, block two, block three, and block four in
Local superpoint, local localization, cloud mapping SLAM (66.7% local). Another client variation is one wherein super point extraction is performed locally as well as localization against a known map. With such a configuration, the only part that is running on the cloud is the map update operation (block four). The cloud component will update the map (using a potentially much larger set of computational resources) and send a version of the map down to the client. This version allows more seamless tracking in the presence of communication channel interruptions.
Local superpoint, cloud localization, cloud mapping SLAM (33.3% local). In this embodiment, while camera capture must be perform locally on the device (i.e., a camera next to the server racks in a datacenter won't help with SLAM), it is possible to only perform a subset of the computations on the local device and the rest in the cloud. In this version of the client, block one and block two (referring to
Cloud-based SLAM (0% local). On another extreme, it is possible to perform all SLAM computations in the cloud (i.e., edge, fog, cloud resources as described above, for brevity in this section, we refer merely to “cloud”) with the device only providing images and a communication channel to the cloud compute resource. With such a thin client configuration, block two, block three, and block four may be performed in the cloud. Block one (image formation and capture) is still performed locally. In order to make such a system real-time (i.e., 30+ frames per second, “fps”), we may need to rapidly encode images and send them to the remote computing resources. The time required to both encode the image into a suitable payload and send the image over the network must be made as small as possible.
Comparison and Bandwidth-requirements. In each of the configurations discussed above that involve a cloud-component, some information from the local device generally must be sent to the cloud. In the case of cloud-slam, one must encode the images and send them to the SLAM system in the cloud. In the case of local super point processing, one may need to send the points and descriptors to the cloud. In the case of a hybrid system with cloud mapping and on-device localization, one may need to send the points, descriptors, and estimated pose to the cloud, but it need not be done at 30+ fps. If the map management is happening in the cloud but a localization module exists locally, then information pertaining to the current map may be periodically sent from the cloud to the client.
Assistance from other sensors and computational resources. The output of other sensors, such as IMUs and depth sensors, can be used to complement superpoint which generally only deals with raw images (color or grayscale). These additional sensors generally must be located on the same device as the physical image sensor. One may refer to these extra bits of information as auxiliary inputs since they are not a hard-requirement for our approach to work. From a computational resource perspective, one may also add additional compute units (such as more CPUs or more GPUs). Additional computational resources may be places far away from the local device, and may be located right in a cloud datacenter, for example. This way, the additional computational resources may be utilized for other tasks when the load is low.
Superpoint on head-mounted displays, smartphones, and other clients. The subject superpoint-based SLAM framework is designed to work across a broad spectrum devices which we refer to as clients. On one extreme, a client can be a barebones image sensor with a Wi-Fi module, and just enough computation to encode and send images over a network. On another extreme, the client may contain multiple cameras, a head-wearable display, additional locally-coupled compute resources, such as edge notes, etc., as described above.
Image-based localization and relocalization across time. By focusing on the machine learning-based extraction and summarization of visual information, the subject approach is designed to be more robust to lighting changes and environmental variations that are typically across 1-2 days in any given environment. This facilitates a SLAM session to persist across multiple days. Using classical image feature extraction procedures, only a small subset of the extracted 2D features are matchable in a tracking scenario. Because RANSAC and other outlier rejection mechanism are heavily used in traditional SLAM methods, those 2D features generally are not very robust for the task of relocalization across large changes in time.
Cross-device localization and relocalization across time. By focusing on images, it may be easier to build a map using one kind of client, and then utilize that map inside another client.
Multi-user localization and mapping. By performing the Cloud Update operations in the cloud, it may be relatively easy to have multiple clients share and update a single 3D map.
The aforementioned configurations facilitate the development and use of spatial computing systems with highly-distributed resource bases, such as those described below, and also above in reference to the various edge, fog, and cloud resource integrations.
Referring to
Thus in these related embodiments, which may also be termed variations on the theme of “Adaptive Neural Computing”, one may pull from many of the same services and sources as in a less connected configuration, and the challenges become more focused on connectivity than on carrying locally all of the requisite computer hardware. With the speeds and bandwidth achievable with IEEE 802.11 ax/ay (i.e., Wi-Fi 6) and 5G, one is able to rethink the ways in which various tasks pertaining to spatial computing may be conducted.
Generally, one of the challenges with Adaptive Neural Computing configurations is to relocate the computational load of the services required to operate the XR device. As described above, in various embodiments, this may be facilitated by creating edge node computational devices and sensors, as well as optimized fog nodes and cloud computing facilities.
An Adaptive Neural Computing edge node may be a comprehensive internet-of-things (i.e., readily connectible using conventional network infrastructure) style device placed at the point of need or the “edge” of compute. Such a device may comprise a small computer capable of high bandwidth connectivity, high speed memory, a GPU, a CPU, and camera interfaces. Suitable edge nodes, as described above, include but are not limited to those marketed by Nvidia™ and Intel™.
Referring to
Such an edge computational device may be configured to be the primary device for low latency operation to facilitate integration with spatial computing systems. For example,
Integrated edge nodes may be configured scale from portable and small up to a full on-premises server capability. As noted above, the purpose of using such a distributed computation across the cloud systems is that at the point of need the lower latency operations will not be performed on a small computer tethered to the wearable.
World reconstruction (i.e., meshing, SLAM) described above in reference to MMO-like configurations with all or most computing resources on board the user's person employ a local-service-based continuous operation configuration. One may utilize an “absolute” coordinate system as a model of the world (the geometry of the digital twin). This model may be fed through a series of transforms until it is laid over the physical environment in a true-scale configuration. These transforms are essentially a calibration file that takes into account the intrinsics and extrinsics of the user's system, the type of device the user is using, or the current state of the user.
Referring to
Once one has this model of the world in an authoritative server, one may implement uploads and downloads on a timetable determined to be most useful from the Digital Twin server to the edge/fog nodes in the local environments. This ensures that the devices are as up to date as is required for a specific application.
Once a coordinate system has been established, it may be populated with data. There are various ways to accomplish this. For example, data from XR devices of various users may be pipelined or included on the local storage of an XR device new to the locale. In another embodiment, various techniques and hardware, such as robotics, sensor-integrated backpacks, and the like may be utilized to provide periodically updated meshing/scanning of various environments. What these data sources do is capture data in the world and using the passable world object recognizers, temporal features, contrasting features, and supervised feature definition we can align the maps and through sensor fusion techniques such as an Extended Kalman-Bucy Filter (EKF) or Feedback Particle Filter (FPF) we can perform continuous time non-linear filtering and align these maps at run-time if needed. This combinatorial map may become the basis for how to perform tasks in spatial computing. The primary goal in this implementation is efficiency, so this means we do not need to constantly reconstruct the world we only need to capture the differences or deltas in the mapped world. In other words, if a mesh of a particular environment is stable, the entire mesh need not be replaced, but only the deltas or changes need to be added.
With all of the raw map data in an authoritative mesh storage device, such as a cloud server, one may now take such data and create an intelligently smoothed and textured view of the world which may be expressed in more detail. This is because it may be desirable to not only want to use this map to align digital content to, but also to recreate the world for virtual reality and pass-through XR technologies. Prior to this, one may contextualize the world to facilitate pre-identification prior to consumption of the data, as well as key features in the raw geometry.
Referring to
A next level may be to identify the semi-permanent structures or things in the room. One implementation accomplishes this with “hierarchal fine-grain segmentation” (i.e., segmenting to smaller and smaller features), which involves semantically labeling features on an object until the features are no longer discernable from one another. An example of this would be to continually segment then label the pieces of a chair.
After understanding the objects in the room, the semi-permanent objects may be identified as those objects, such as a large dining room table, which can move but probably will not frequently move, and/or will take a significant effort to move. Next, one may identify objects that are dynamically moving, such as laptops or clouds in the sky. This may be accomplished with object level scene reconstruction with proactive object analysis, for example. By constantly scanning a room and looking at the dynamic objects, the system may be configured to be able to increase the probability of proper segmentation.
Referring back to
Once the room has been segmented, labeled and contextualized, one may turn to connecting the room to digital “sockets”—which is a metaphor that may be used for forming intersections between a digital world and the physical world. A digital socket may be an object, device, robot, or anything that plugs the digital world to the physical world. Intent to interact with the underlying digital world and how it effects the physical world may be conveyed through these sockets, such as by meta-data to each particular socket. If the physical world is too dark for example, the user interfaces for XR devices to utilize such socket may be paired with an internet-of-things (“IoT”) controller integrated with an application and user interface (“UI”) which changes the light settings of the device. As another example, when a particular remotely-located XR device user is considering which port or socket to join as he wants to view a particular building, he may choose the socket which, based upon the meta data, will take him virtually straight into a very well-meshed room with a significant amount of pre-existing data pertaining to all features of the room, in full color. This interaction may require many processes, and the combination of such processes into a tool which a user can use to change their understanding of a perceived world.
One now may have a fully contextualized and robust digital twin of the physical world. Referring to
Pose-related data may be streamed from the user's XR device, or from other XR devices, to available edge nodes, fog nodes, or to the cloud. As noted above, such distribution may be configured to only require a small amount of information, and in times where the computational load is low, the device can stream images back to the edge node to ensure that the localization has not drifted.
In
The second row in
A next element in this implementation of spatial computing is the route taken to render objects to the XR device. Since these devices can have multiple types of operating systems, one may adapt a streaming or Edge rendering protocol configured to take advantage of the spatial nature of the digital world and physical world, with distributed compute resources and various levels of latency, as described above. This being said, generally it is known where the device is in the room based on the pose methodology described above. The room is also known, due to the reconstructed world.
Edge Rendering may be facilitated by modern connectivity availability and should become more and more prevalent as more and more XR systems become utilized. Remote rendering can be accomplished through many variations of classical render pipelines, but generally will provide more latency than is acceptable in most spatial computing configurations to effectively communicate the inferred data.
One method for accomplishing remote rendering for spatial computing systems is to take advantage of conventional rendering and streaming techniques utilized by companies who stream movies or tv shows (i.e., such as Netflix™). In other words, a system may be utilized to implement the current render pipeline on distributed edge/fog/cloud resources, and then take this output, convert the 3D content to 2D video, and stream that to the device.
As noted above, for the configurations described herein one may take advantage of recent developments in wireless connectivity, including but not limited to WiFi-6, which also may be known as IEEE 802.11ax standards compliant, or any successor which will be able to effectively transmit the signals to the device.
One overall schema for rendering on a particular XR device may comprise computing all of the relevant processes in the distributed cloud, streaming the results to the XR device—directly to the frame buffer of said device, and producing imagery on the display for the user.
The aforementioned superpoint techniques may be utilized in aggregating meshes, or forming geometric relationships between them.
The physical world is not only an interaction element in spatial computing—it is also one of the primary inputs for the computational architecture. In order for this simplification of data to occur we need to understand the environment in which a user or experience (Location Based Experience LBE) is located in.
Once one has raw information from the sensor or sensors, either at run-time, or saved and then processed, or some combination thereof, one may find the points of interest. As noted above one may use superpoint techniques (504), which employ a self-supervised interest point detector and descriptor, thus allowing for features of interest to be identified.
To optimize ingesting or processing data from multiple sources into a single mesh (506) which is the digital representation of the world, one may need to conduct a few backend processes (512, 514, 516, 518). One may be utilized to populate any new area of the world into what may be termed an “Authoritative Intelligent Mesh Server” (or “AIMS”), which also may be termed a “Spatial Atlas”. If data does exist, then one may perform sensor fusion to combine the information into one single mesh. Sensor fusion of this type can be done in traditional methods such as the many variations of Kalman Filters, or we can use Deep Learning techniques to use the found superpoint features of interest and conduct feature level fusion for each one of the sensor types and formats, by using the superpoint fully-convolutional neural network architecture we can create synthetic datasets with each type of sensor and then use that to train each of the individual CNNs. Once the superpoint algorithm has been tuned for each sensor type, a feature level fusion may occur following the general pattern of the image below, where one has implemented a mode-specific neural network.
Once the unified mesh is calculated we are faced with the challenge of what to do with the mesh. In one framework, one may seek further contextualization, in another the system may be configured to create an intelligent mesh (507). A superpoint intelligent mesh may employ using a superpoint algorithm and taking advantage of the homographic adaption implemented to create an intelligent interpolation of the primary mesh, and adding data points by enabling larger probability of additional features. In order to accomplish this one may follow the same process as with a superpoint feature detector, thereby increasing the training set to include the three dimensional mesh of the 2D shapes in the original superpoint. The reason this may be needed is after the mesh unification is that the same superpoints may not be entirely consistent and one may want all features in the frame to be extracted as the system is going to use them to overlay (508) the texture on the wireframe. A homographic adaptation of the feature plane may result in one set of superpoints, and since the entire map is in 3D space, one can rotate the user perspective as well about the super point along the arc created by the depth from the user to that point to create more perspectives of the features identified by superpoint. We then will use all of the points we create to attach the texture (510) to the surface of the 3D map and pseudo-depth-image.
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Various example embodiments of the invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate more broadly applicable aspects of the invention. Various changes may be made to the invention described and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act(s) or step(s) to the objective(s), spirit or scope of the present invention. Further, as will be appreciated by those with skill in the art that each of the individual variations described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present inventions. All such modifications are intended to be within the scope of claims associated with this disclosure.
The invention includes methods that may be performed using the subject devices. The methods may comprise the act of providing such a suitable device. Such provision may be performed by the end user. In other words, the “providing” act merely requires the end user obtain, access, approach, position, set-up, activate, power-up or otherwise act to provide the requisite device in the subject method. Methods recited herein may be carried out in any order of the recited events which is logically possible, as well as in the recited order of events.
Example aspects of the invention, together with details regarding material selection and manufacture have been set forth above. As for other details of the present invention, these may be appreciated in connection with the above-referenced patents and publications as well as generally known or appreciated by those with skill in the art. The same may hold true with respect to method-based aspects of the invention in terms of additional acts as commonly or logically employed.
In addition, though the invention has been described in reference to several examples optionally incorporating various features, the invention is not to be limited to that which is described or indicated as contemplated with respect to each variation of the invention. Various changes may be made to the invention described and equivalents (whether recited herein or not included for the sake of some brevity) may be substituted without departing from the true spirit and scope of the invention. In addition, where a range of values is provided, it is understood that every intervening value, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the invention.
Also, it is contemplated that any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein. Reference to a singular item, includes the possibility that there are plural of the same items present. More specifically, as used herein and in claims associated hereto, the singular forms “a,” “an,” “said,” and “the” include plural referents unless the specifically stated otherwise. In other words, use of the articles allow for “at least one” of the subject item in the description above as well as claims associated with this disclosure. It is further noted that such claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.
Without the use of such exclusive terminology, the term “comprising” in claims associated with this disclosure shall allow for the inclusion of any additional element—irrespective of whether a given number of elements are enumerated in such claims, or the addition of a feature could be regarded as transforming the nature of an element set forth in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given as broad a commonly understood meaning as possible while maintaining claim validity.
The breadth of the present invention is not to be limited to the examples provided and/or the subject specification, but rather only by the scope of claim language associated with this disclosure.
While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the current invention, and that this invention is not restricted to the specific constructions and arrangements shown and described since modifications may occur to those ordinarily skilled in the art.
This is a divisional application of U.S. patent application Ser. No. 16/864,721, filed on May 1, 2020, which claims priority from U.S. Provisional Patent Application No. 62/841,806, filed on May 1, 2019, all of which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4344092 | Miller | Aug 1982 | A |
| 4652930 | Crawford | Mar 1987 | A |
| 4810080 | Grendol et al. | Mar 1989 | A |
| 4997268 | Dauvergne | Mar 1991 | A |
| 5007727 | Kahaney et al. | Apr 1991 | A |
| 5074295 | Willis | Dec 1991 | A |
| 5240220 | Elberbaum | Aug 1993 | A |
| 5251635 | Dumoulin et al. | Oct 1993 | A |
| 5410763 | Bolle | May 1995 | A |
| 5455625 | Englander | Oct 1995 | A |
| 5495286 | Adair | Feb 1996 | A |
| 5497463 | Stein et al. | Mar 1996 | A |
| 5659701 | Amit et al. | Aug 1997 | A |
| 5682255 | Friesem et al. | Oct 1997 | A |
| 5689669 | Lynch | Nov 1997 | A |
| 5689835 | Chao | Nov 1997 | A |
| 5826092 | Flannery | Oct 1998 | A |
| 5854872 | Tai | Dec 1998 | A |
| 5864365 | Sramek et al. | Jan 1999 | A |
| 5937202 | Crosetto | Aug 1999 | A |
| 6002853 | de Hond | Dec 1999 | A |
| 6012811 | Chao et al. | Jan 2000 | A |
| 6016160 | Coombs et al. | Jan 2000 | A |
| 6064749 | Hirota et al. | May 2000 | A |
| 6076927 | Owens | Jun 2000 | A |
| 6079982 | Meader | Jun 2000 | A |
| 6117923 | Amagai et al. | Sep 2000 | A |
| 6119147 | Toomey | Sep 2000 | A |
| 6124977 | Takahashi | Sep 2000 | A |
| 6179619 | Tanaka | Jan 2001 | B1 |
| 6191809 | Hori et al. | Feb 2001 | B1 |
| 6219045 | Leahy | Apr 2001 | B1 |
| 6243091 | Berstis | Jun 2001 | B1 |
| 6271843 | Lection | Aug 2001 | B1 |
| 6362817 | Powers | Mar 2002 | B1 |
| 6375369 | Schneider et al. | Apr 2002 | B1 |
| 6385735 | Wilson | May 2002 | B1 |
| 6396522 | Vu | May 2002 | B1 |
| 6414679 | Miodonski | Jul 2002 | B1 |
| 6538655 | Kubota | Mar 2003 | B1 |
| 6541736 | Huang et al. | Apr 2003 | B1 |
| 6570563 | Honda | May 2003 | B1 |
| 6573903 | Gantt | Jun 2003 | B2 |
| 6590593 | Robertson | Jul 2003 | B1 |
| 6621508 | Shiraishi | Sep 2003 | B1 |
| 6690393 | Heron | Feb 2004 | B2 |
| 6757068 | Foxlin | Jun 2004 | B2 |
| 6784901 | Harvey | Aug 2004 | B1 |
| 6961055 | Doak | Nov 2005 | B2 |
| 7046515 | Wyatt | May 2006 | B1 |
| 7051219 | Hwang | May 2006 | B2 |
| 7076674 | Cervantes | Jul 2006 | B2 |
| 7111290 | Yates, Jr | Sep 2006 | B1 |
| 7119819 | Robertson | Oct 2006 | B1 |
| 7211986 | Flowerdew et al. | May 2007 | B1 |
| 7219245 | Raghuvanshi | May 2007 | B1 |
| 7382288 | Wilson | Jun 2008 | B1 |
| 7414629 | Santodomingo | Aug 2008 | B2 |
| 7431453 | Hogan | Oct 2008 | B2 |
| 7467356 | Gettman | Dec 2008 | B2 |
| 7542040 | Templeman | Jun 2009 | B2 |
| 7573640 | Nivon et al. | Aug 2009 | B2 |
| 7653877 | Matsuda | Jan 2010 | B2 |
| 7663625 | Chartier | Feb 2010 | B2 |
| 7724980 | Shenzhi | May 2010 | B1 |
| 7746343 | Charaniya | Jun 2010 | B1 |
| 7751662 | Kleemann | Jul 2010 | B2 |
| 7758185 | Lewis | Jul 2010 | B2 |
| 7788323 | Greenstein | Aug 2010 | B2 |
| 7804507 | Yang | Sep 2010 | B2 |
| 7814429 | Buffet | Oct 2010 | B2 |
| 7817150 | Reichard | Oct 2010 | B2 |
| 7844724 | Van Wie | Nov 2010 | B2 |
| 8060759 | Arnan et al. | Nov 2011 | B1 |
| 8120851 | Iwasa | Feb 2012 | B2 |
| 8214660 | Capps, Jr. | Jul 2012 | B2 |
| 8246408 | Elliot | Aug 2012 | B2 |
| 8353594 | Lewis | Jan 2013 | B2 |
| 8360578 | Nummela et al. | Jan 2013 | B2 |
| 8408696 | Hsieh | Apr 2013 | B2 |
| 8508676 | Silverstein et al. | Aug 2013 | B2 |
| 8547638 | Levola | Oct 2013 | B2 |
| 8605764 | Rothaar et al. | Oct 2013 | B1 |
| 8619365 | Harris et al. | Dec 2013 | B2 |
| 8696113 | Lewis | Apr 2014 | B2 |
| 8698701 | Margulis | Apr 2014 | B2 |
| 8733927 | Lewis | May 2014 | B1 |
| 8736636 | Kang | May 2014 | B2 |
| 8759929 | Shiozawa et al. | Jun 2014 | B2 |
| 8793770 | Lim | Jul 2014 | B2 |
| 8823855 | Hwang | Sep 2014 | B2 |
| 8847988 | Geisner et al. | Sep 2014 | B2 |
| 8874673 | Kim | Oct 2014 | B2 |
| 9010929 | Lewis | Apr 2015 | B2 |
| 9015501 | Gee | Apr 2015 | B2 |
| 9086537 | Iwasa et al. | Jul 2015 | B2 |
| 9095437 | Boyden et al. | Aug 2015 | B2 |
| 9239473 | Lewis | Jan 2016 | B2 |
| 9244293 | Lewis | Jan 2016 | B2 |
| 9244533 | Friend | Jan 2016 | B2 |
| 9285872 | Raffle et al. | Mar 2016 | B1 |
| 9383823 | Geisner et al. | Jul 2016 | B2 |
| 9489027 | Ogletree | Nov 2016 | B1 |
| 9519305 | Wolfe | Dec 2016 | B2 |
| 9581820 | Robbins | Feb 2017 | B2 |
| 9582060 | Balatsos | Feb 2017 | B2 |
| 9658473 | Lewis | May 2017 | B2 |
| 9671566 | Abovitz et al. | Jun 2017 | B2 |
| 9671615 | Vallius et al. | Jun 2017 | B1 |
| 9696795 | Marcolina | Jul 2017 | B2 |
| 9798144 | Sako et al. | Oct 2017 | B2 |
| 9874664 | Stevens et al. | Jan 2018 | B2 |
| 9880441 | Osterhout | Jan 2018 | B1 |
| 9918058 | Takahasi et al. | Mar 2018 | B2 |
| 9955862 | Freeman et al. | May 2018 | B2 |
| 9978118 | Ozgumer et al. | May 2018 | B1 |
| 9996797 | Holz | Jun 2018 | B1 |
| 10018844 | Levola et al. | Jul 2018 | B2 |
| 10082865 | Raynal et al. | Sep 2018 | B1 |
| 10151937 | Lewis | Dec 2018 | B2 |
| 10185147 | Lewis | Jan 2019 | B2 |
| 10218679 | Jawahar | Feb 2019 | B1 |
| 10241545 | Richards et al. | Mar 2019 | B1 |
| 10317680 | Richards et al. | Jun 2019 | B1 |
| 10436594 | Belt et al. | Oct 2019 | B2 |
| 10516853 | Gibson | Dec 2019 | B1 |
| 10551879 | Richards et al. | Feb 2020 | B1 |
| 10578870 | Kimmel | Mar 2020 | B2 |
| 10646283 | Johnson et al. | May 2020 | B2 |
| 10698202 | Kimmel et al. | Jun 2020 | B2 |
| 10856107 | Mycek et al. | Oct 2020 | B2 |
| 10825424 | Zhang | Nov 2020 | B2 |
| 10987176 | Poltaretskyi et al. | Apr 2021 | B2 |
| 11190681 | Brook et al. | Nov 2021 | B1 |
| 11209656 | Choi et al. | Dec 2021 | B1 |
| 11236993 | Hall et al. | Feb 2022 | B1 |
| 11710430 | Wray | Jul 2023 | B1 |
| 20010010598 | Aritake et al. | Aug 2001 | A1 |
| 20010018667 | Kim | Aug 2001 | A1 |
| 20020007463 | Fung | Jan 2002 | A1 |
| 20020108064 | Nunally | Feb 2002 | A1 |
| 20020063913 | Nakamura et al. | May 2002 | A1 |
| 20020071050 | Homberg | Jun 2002 | A1 |
| 20020095463 | Matsuda | Jul 2002 | A1 |
| 20020113820 | Robinson | Aug 2002 | A1 |
| 20020122648 | Mule' et al. | Sep 2002 | A1 |
| 20020140848 | Cooper et al. | Oct 2002 | A1 |
| 20030028816 | Bacon | Feb 2003 | A1 |
| 20030048456 | Hill | Mar 2003 | A1 |
| 20030067685 | Niv | Apr 2003 | A1 |
| 20030077458 | Korenaga et al. | Apr 2003 | A1 |
| 20030080976 | Satoh et al. | May 2003 | A1 |
| 20030115494 | Cervantes | Jun 2003 | A1 |
| 20030218614 | Lavelle et al. | Nov 2003 | A1 |
| 20030219992 | Schaper | Nov 2003 | A1 |
| 20030226047 | Park | Dec 2003 | A1 |
| 20040001533 | Tran et al. | Jan 2004 | A1 |
| 20040021600 | Wittenberg | Feb 2004 | A1 |
| 20040025069 | Gary et al. | Feb 2004 | A1 |
| 20040042377 | Nikoloai et al. | Mar 2004 | A1 |
| 20040073822 | Greco | Apr 2004 | A1 |
| 20040073825 | Itoh | Apr 2004 | A1 |
| 20040111248 | Granny et al. | Jun 2004 | A1 |
| 20040113887 | Pair | Jun 2004 | A1 |
| 20040174496 | Ji et al. | Sep 2004 | A1 |
| 20040186902 | Stewart | Sep 2004 | A1 |
| 20040193441 | Altieri | Sep 2004 | A1 |
| 20040201857 | Foxlin | Oct 2004 | A1 |
| 20040238732 | State et al. | Dec 2004 | A1 |
| 20040240072 | Schindler et al. | Dec 2004 | A1 |
| 20040246391 | Travis | Dec 2004 | A1 |
| 20040268159 | Aasheim et al. | Dec 2004 | A1 |
| 20050001977 | Zelman | Jan 2005 | A1 |
| 20050034002 | Flautner | Feb 2005 | A1 |
| 20050093719 | Okamoto | May 2005 | A1 |
| 20050128212 | Edecker | Jun 2005 | A1 |
| 20050157159 | Komiya et al. | Jul 2005 | A1 |
| 20050177385 | Hull | Aug 2005 | A1 |
| 20050231599 | Yamasaki | Oct 2005 | A1 |
| 20050273792 | Inohara et al. | Dec 2005 | A1 |
| 20060013435 | Rhoads | Jan 2006 | A1 |
| 20060015821 | Jacques Parker et al. | Jan 2006 | A1 |
| 20060019723 | Vorenkamp | Jan 2006 | A1 |
| 20060038880 | Starkweather et al. | Feb 2006 | A1 |
| 20060050224 | Smith | Mar 2006 | A1 |
| 20060090092 | Verhulst | Apr 2006 | A1 |
| 20060126181 | Levola | Jun 2006 | A1 |
| 20060126182 | Tapani | Jun 2006 | A1 |
| 20060129852 | Bonola | Jun 2006 | A1 |
| 20060132914 | Weiss et al. | Jun 2006 | A1 |
| 20060179329 | Terechko | Aug 2006 | A1 |
| 20060221448 | Nivon et al. | Oct 2006 | A1 |
| 20060228073 | Mukawa et al. | Oct 2006 | A1 |
| 20060250322 | Hall et al. | Nov 2006 | A1 |
| 20060259621 | Ranganathan | Nov 2006 | A1 |
| 20060268220 | Hogan | Nov 2006 | A1 |
| 20070058248 | Nguyen et al. | Mar 2007 | A1 |
| 20070103836 | Oh | May 2007 | A1 |
| 20070124730 | Pytel | May 2007 | A1 |
| 20070159673 | Freeman et al. | Jul 2007 | A1 |
| 20070185398 | Kimura et al. | Aug 2007 | A1 |
| 20070188837 | Shimizu et al. | Aug 2007 | A1 |
| 20070198886 | Saito | Aug 2007 | A1 |
| 20070204672 | Huang et al. | Sep 2007 | A1 |
| 20070213952 | Cirelli | Sep 2007 | A1 |
| 20070283247 | Brenneman et al. | Dec 2007 | A1 |
| 20080002259 | Ishizawa et al. | Jan 2008 | A1 |
| 20080002260 | Arrouy et al. | Jan 2008 | A1 |
| 20080030429 | Hailpern | Feb 2008 | A1 |
| 20080043334 | Itzkovitch et al. | Feb 2008 | A1 |
| 20080046773 | Ham | Feb 2008 | A1 |
| 20080063802 | Maula et al. | Mar 2008 | A1 |
| 20080068557 | Menduni et al. | Mar 2008 | A1 |
| 20080084533 | Jannard et al. | Apr 2008 | A1 |
| 20080125218 | Collins | May 2008 | A1 |
| 20080146942 | Dala-Krishna | Jun 2008 | A1 |
| 20080173036 | Willaims | Jul 2008 | A1 |
| 20080177506 | Kim | Jul 2008 | A1 |
| 20080183190 | Adcox et al. | Jul 2008 | A1 |
| 20080205838 | Crippa et al. | Aug 2008 | A1 |
| 20080215907 | Wilson | Sep 2008 | A1 |
| 20080225393 | Rinko | Sep 2008 | A1 |
| 20080235570 | Sawada | Sep 2008 | A1 |
| 20080246693 | Hailpern | Oct 2008 | A1 |
| 20080316768 | Travis | Dec 2008 | A1 |
| 20090076791 | Rhoades | Mar 2009 | A1 |
| 20090091583 | McCoy | Apr 2009 | A1 |
| 20090153797 | Allon et al. | Jun 2009 | A1 |
| 20090177445 | Capps, Jr. et al. | Jul 2009 | A1 |
| 20090224416 | Laakkonen et al. | Sep 2009 | A1 |
| 20090245730 | Kleemann | Oct 2009 | A1 |
| 20090287728 | Martine | Nov 2009 | A1 |
| 20090300528 | Stambaugh | Dec 2009 | A1 |
| 20090310633 | Ikegami | Dec 2009 | A1 |
| 20100005326 | Archer | Jan 2010 | A1 |
| 20100019962 | Fujita | Jan 2010 | A1 |
| 20100056274 | Uusitalo et al. | Mar 2010 | A1 |
| 20100060970 | Harris et al. | Mar 2010 | A1 |
| 20100063854 | Purvis et al. | Mar 2010 | A1 |
| 20100070378 | Trotman | Mar 2010 | A1 |
| 20100079841 | Levola | Apr 2010 | A1 |
| 20100115428 | Shuping | May 2010 | A1 |
| 20100153934 | Lachner | Jun 2010 | A1 |
| 20100194632 | Raento et al. | Aug 2010 | A1 |
| 20100205541 | Rapaport | Aug 2010 | A1 |
| 20100214284 | Rieffel | Aug 2010 | A1 |
| 20100232016 | Landa et al. | Sep 2010 | A1 |
| 20100232031 | Batchko et al. | Sep 2010 | A1 |
| 20100244168 | Shiozawa et al. | Sep 2010 | A1 |
| 20100274567 | Carlson | Oct 2010 | A1 |
| 20100274627 | Carlson | Oct 2010 | A1 |
| 20100277803 | Pockett et al. | Nov 2010 | A1 |
| 20100284085 | Laakkonen | Nov 2010 | A1 |
| 20100287485 | Bertolami et al. | Nov 2010 | A1 |
| 20100296163 | Sarikko | Nov 2010 | A1 |
| 20100306715 | Geisner et al. | Dec 2010 | A1 |
| 20100309687 | Sampsell et al. | Dec 2010 | A1 |
| 20110010636 | Hamilton, II | Jan 2011 | A1 |
| 20110021263 | Anderson et al. | Jan 2011 | A1 |
| 20110022870 | Mcgrane | Jan 2011 | A1 |
| 20110041083 | Gabai | Feb 2011 | A1 |
| 20110050640 | Lundback et al. | Mar 2011 | A1 |
| 20110050655 | Mukawa | Mar 2011 | A1 |
| 20110064268 | Cobb et al. | Mar 2011 | A1 |
| 20110122240 | Becker | May 2011 | A1 |
| 20110145617 | Thomson et al. | Jun 2011 | A1 |
| 20110170801 | Lu et al. | Jul 2011 | A1 |
| 20110218733 | Hamza et al. | Sep 2011 | A1 |
| 20110286735 | Temblay | Nov 2011 | A1 |
| 20110291969 | Rashid et al. | Dec 2011 | A1 |
| 20120011389 | Driesen | Jan 2012 | A1 |
| 20120050535 | Densham et al. | Mar 2012 | A1 |
| 20120075501 | Oyagi et al. | Mar 2012 | A1 |
| 20120079466 | Gonion | Mar 2012 | A1 |
| 20120081392 | Arthur | Apr 2012 | A1 |
| 20120089854 | Breakstone | Apr 2012 | A1 |
| 20120113235 | Shintani | May 2012 | A1 |
| 20120127062 | Bar-Zeev et al. | May 2012 | A1 |
| 20120154557 | Perez et al. | Jun 2012 | A1 |
| 20120215094 | Rahimian et al. | Aug 2012 | A1 |
| 20120218301 | Miller | Aug 2012 | A1 |
| 20120246506 | Knight | Sep 2012 | A1 |
| 20120249416 | Maciocci et al. | Oct 2012 | A1 |
| 20120249741 | Maciocci | Oct 2012 | A1 |
| 20120260083 | Andrews | Oct 2012 | A1 |
| 20120307075 | Margalit | Dec 2012 | A1 |
| 20120307362 | Silverstein et al. | Dec 2012 | A1 |
| 20120314959 | White et al. | Dec 2012 | A1 |
| 20120320460 | Levola | Dec 2012 | A1 |
| 20120326948 | Crocco et al. | Dec 2012 | A1 |
| 20130021486 | Richardon | Jan 2013 | A1 |
| 20130050258 | Liu et al. | Feb 2013 | A1 |
| 20130050642 | Lewis et al. | Feb 2013 | A1 |
| 20130050833 | Lewis et al. | Feb 2013 | A1 |
| 20130051730 | Travers et al. | Feb 2013 | A1 |
| 20130061240 | Yan et al. | Mar 2013 | A1 |
| 20130077049 | Bohn | Mar 2013 | A1 |
| 20130077170 | Ukuda | Mar 2013 | A1 |
| 20130094148 | Sloane | Apr 2013 | A1 |
| 20130129282 | Li | May 2013 | A1 |
| 20130162940 | Kurtin et al. | Jun 2013 | A1 |
| 20130169923 | Schnoll et al. | Jul 2013 | A1 |
| 20130205126 | Kruglick | Aug 2013 | A1 |
| 20130222386 | Tannhauser et al. | Aug 2013 | A1 |
| 20130268257 | Hu | Oct 2013 | A1 |
| 20130278633 | Ahn et al. | Oct 2013 | A1 |
| 20130314789 | Saarikko et al. | Nov 2013 | A1 |
| 20130318276 | Dalal | Nov 2013 | A1 |
| 20130336138 | Venkatraman et al. | Dec 2013 | A1 |
| 20130342564 | Kinnebrew et al. | Dec 2013 | A1 |
| 20130342570 | Kinnebrew et al. | Dec 2013 | A1 |
| 20130342571 | Kinnebrew et al. | Dec 2013 | A1 |
| 20130343408 | Cook | Dec 2013 | A1 |
| 20140002329 | Nishimaki et al. | Jan 2014 | A1 |
| 20140013098 | Yeung | Jan 2014 | A1 |
| 20140016821 | Arth et al. | Jan 2014 | A1 |
| 20140022819 | Oh et al. | Jan 2014 | A1 |
| 20140078023 | Ikeda et al. | Mar 2014 | A1 |
| 20140082526 | Park | Mar 2014 | A1 |
| 20140119598 | Ramachandran et al. | May 2014 | A1 |
| 20140123015 | Sako et al. | May 2014 | A1 |
| 20140126769 | Reitmayr et al. | May 2014 | A1 |
| 20140140653 | Brown et al. | May 2014 | A1 |
| 20140149573 | Tofighbakhsh et al. | May 2014 | A1 |
| 20140168260 | O'Brien et al. | Jun 2014 | A1 |
| 20140204438 | Yamada et al. | Jul 2014 | A1 |
| 20140244983 | Mcdonald et al. | Aug 2014 | A1 |
| 20140266987 | Magyari | Sep 2014 | A1 |
| 20140267419 | Ballard et al. | Sep 2014 | A1 |
| 20140274391 | Stafford | Sep 2014 | A1 |
| 20140282105 | Nordstrom | Sep 2014 | A1 |
| 20140292645 | Tsurumi et al. | Oct 2014 | A1 |
| 20140309031 | Suzuki | Oct 2014 | A1 |
| 20140313228 | Kasahara | Oct 2014 | A1 |
| 20140333612 | Itoh et al. | Nov 2014 | A1 |
| 20140340449 | Plagemann et al. | Nov 2014 | A1 |
| 20140359589 | Kodsky et al. | Dec 2014 | A1 |
| 20140375680 | Ackerman et al. | Dec 2014 | A1 |
| 20150005785 | Olson | Jan 2015 | A1 |
| 20150009099 | Queen | Jan 2015 | A1 |
| 20150015842 | Chen | Jan 2015 | A1 |
| 20150077312 | Wang | Mar 2015 | A1 |
| 20150097719 | Balachandreswaran et al. | Apr 2015 | A1 |
| 20150123966 | Newman | May 2015 | A1 |
| 20150130790 | Vasquez, II et al. | May 2015 | A1 |
| 20150134995 | Park et al. | May 2015 | A1 |
| 20150138248 | Schrader | May 2015 | A1 |
| 20150155939 | Oshima et al. | Jun 2015 | A1 |
| 20150168221 | Mao et al. | Jun 2015 | A1 |
| 20150205126 | Schowengerdt | Jul 2015 | A1 |
| 20150235427 | Nobori et al. | Aug 2015 | A1 |
| 20150235431 | Schowengerdt | Aug 2015 | A1 |
| 20150253651 | Russell et al. | Sep 2015 | A1 |
| 20150256484 | Cameron | Sep 2015 | A1 |
| 20150269784 | Miyawaki et al. | Sep 2015 | A1 |
| 20150294483 | Wells et al. | Oct 2015 | A1 |
| 20150301955 | Yakovenko et al. | Oct 2015 | A1 |
| 20150310657 | Eden | Oct 2015 | A1 |
| 20150338915 | Publicover et al. | Nov 2015 | A1 |
| 20150355481 | Hilkes et al. | Dec 2015 | A1 |
| 20160004102 | Nisper et al. | Jan 2016 | A1 |
| 20160015470 | Border | Jan 2016 | A1 |
| 20160027215 | Burns et al. | Jan 2016 | A1 |
| 20160033770 | Fujimaki et al. | Feb 2016 | A1 |
| 20160051217 | Douglas et al. | Feb 2016 | A1 |
| 20160077338 | Robbins et al. | Mar 2016 | A1 |
| 20160085285 | Mangione-Smith | Mar 2016 | A1 |
| 20160085300 | Robbins et al. | Mar 2016 | A1 |
| 20160091720 | Stafford et al. | Mar 2016 | A1 |
| 20160093099 | Bridges | Mar 2016 | A1 |
| 20160093269 | Buckley et al. | Mar 2016 | A1 |
| 20160103326 | Kimura et al. | Apr 2016 | A1 |
| 20160123745 | Cotier et al. | May 2016 | A1 |
| 20160139402 | Lapstun | May 2016 | A1 |
| 20160139411 | Kang et al. | May 2016 | A1 |
| 20160155273 | Lyren et al. | Jun 2016 | A1 |
| 20160163063 | Ashman | Jun 2016 | A1 |
| 20160180596 | Gonzalez del Rosario | Jun 2016 | A1 |
| 20160187654 | Border et al. | Jun 2016 | A1 |
| 20160191887 | Casas | Jun 2016 | A1 |
| 20160202496 | Billetz et al. | Jul 2016 | A1 |
| 20160217624 | Finn et al. | Jul 2016 | A1 |
| 20160266412 | Yoshida | Sep 2016 | A1 |
| 20160267708 | Nistico et al. | Sep 2016 | A1 |
| 20160274733 | Hasegawa et al. | Sep 2016 | A1 |
| 20160287337 | Aram et al. | Oct 2016 | A1 |
| 20160300388 | Stafford et al. | Oct 2016 | A1 |
| 20160321551 | Priness et al. | Nov 2016 | A1 |
| 20160327798 | Xiao et al. | Nov 2016 | A1 |
| 20160334279 | Mittleman et al. | Nov 2016 | A1 |
| 20160357255 | Lindh et al. | Dec 2016 | A1 |
| 20160370404 | Quadrat et al. | Dec 2016 | A1 |
| 20160370510 | Thomas | Dec 2016 | A1 |
| 20170038607 | Camara | Feb 2017 | A1 |
| 20170060225 | Zha et al. | Mar 2017 | A1 |
| 20170061696 | Li et al. | Mar 2017 | A1 |
| 20170064066 | Das et al. | Mar 2017 | A1 |
| 20170100664 | Osterhout et al. | Apr 2017 | A1 |
| 20170102544 | Vallius et al. | Apr 2017 | A1 |
| 20170115487 | Travis | Apr 2017 | A1 |
| 20170122725 | Yeoh et al. | May 2017 | A1 |
| 20170123526 | Trail et al. | May 2017 | A1 |
| 20170127295 | Black et al. | May 2017 | A1 |
| 20170131569 | Aschwanden et al. | May 2017 | A1 |
| 20170147066 | Katz et al. | May 2017 | A1 |
| 20170160518 | Lanman et al. | Jun 2017 | A1 |
| 20170161951 | Fix et al. | Jun 2017 | A1 |
| 20170172409 | Cavin et al. | Jun 2017 | A1 |
| 20170185261 | Perez | Jun 2017 | A1 |
| 20170192239 | Nakamura et al. | Jul 2017 | A1 |
| 20170201709 | Igarashi et al. | Jul 2017 | A1 |
| 20170205903 | Miller et al. | Jul 2017 | A1 |
| 20170206668 | Poulos et al. | Jul 2017 | A1 |
| 20170213388 | Margolis et al. | Jul 2017 | A1 |
| 20170214907 | Lapstun | Jul 2017 | A1 |
| 20170219841 | Popovich et al. | Aug 2017 | A1 |
| 20170232345 | Rofougaran et al. | Aug 2017 | A1 |
| 20170235126 | DiDomenico | Aug 2017 | A1 |
| 20170235129 | Kamakura | Aug 2017 | A1 |
| 20170235142 | Wall et al. | Aug 2017 | A1 |
| 20170235144 | Piskunov et al. | Aug 2017 | A1 |
| 20170235147 | Kamakura | Aug 2017 | A1 |
| 20170243403 | Daniels | Aug 2017 | A1 |
| 20170246070 | Osterhout et al. | Aug 2017 | A1 |
| 20170254832 | Ho et al. | Sep 2017 | A1 |
| 20170256096 | Faaborg et al. | Sep 2017 | A1 |
| 20170258526 | Lang | Sep 2017 | A1 |
| 20170266529 | Reikmoto | Sep 2017 | A1 |
| 20170270712 | Tyson et al. | Sep 2017 | A1 |
| 20170281054 | Stever et al. | Oct 2017 | A1 |
| 20170287376 | Bakar et al. | Oct 2017 | A1 |
| 20170293141 | Schowengerdt et al. | Oct 2017 | A1 |
| 20170307886 | Stenberg et al. | Oct 2017 | A1 |
| 20170307891 | Bucknor et al. | Oct 2017 | A1 |
| 20170312032 | Amanatullah et al. | Nov 2017 | A1 |
| 20170322418 | Liu et al. | Nov 2017 | A1 |
| 20170322426 | Tervo | Nov 2017 | A1 |
| 20170329137 | Tervo | Nov 2017 | A1 |
| 20170332098 | Rusanovskyy et al. | Nov 2017 | A1 |
| 20170336636 | Amitai et al. | Nov 2017 | A1 |
| 20170336867 | Wilairat et al. | Nov 2017 | A1 |
| 20170357332 | Balan et al. | Dec 2017 | A1 |
| 20170363871 | Vallius | Dec 2017 | A1 |
| 20170371394 | Chan | Dec 2017 | A1 |
| 20170371661 | Sparling | Dec 2017 | A1 |
| 20180014266 | Chen | Jan 2018 | A1 |
| 20180024289 | Fattal | Jan 2018 | A1 |
| 20180039673 | Hen et al. | Feb 2018 | A1 |
| 20180044173 | Netzer | Feb 2018 | A1 |
| 20180052007 | Teskey et al. | Feb 2018 | A1 |
| 20180052501 | Jones, Jr. et al. | Feb 2018 | A1 |
| 20180056305 | Sankey et al. | Mar 2018 | A1 |
| 20180059305 | Popovich et al. | Mar 2018 | A1 |
| 20180067779 | Pillalamarri et al. | Mar 2018 | A1 |
| 20180070855 | Eichler | Mar 2018 | A1 |
| 20180082480 | White et al. | Mar 2018 | A1 |
| 20180084245 | Lapstun | Mar 2018 | A1 |
| 20180088185 | Woods et al. | Mar 2018 | A1 |
| 20180101214 | Mahindru et al. | Apr 2018 | A1 |
| 20180102981 | Kurtzman et al. | Apr 2018 | A1 |
| 20180108179 | Tomlin et al. | Apr 2018 | A1 |
| 20180114298 | Malaika et al. | Apr 2018 | A1 |
| 20180129112 | Osterhout | May 2018 | A1 |
| 20180131907 | Schmirler | May 2018 | A1 |
| 20180136466 | Ko | May 2018 | A1 |
| 20180144691 | Choi et al. | May 2018 | A1 |
| 20180150971 | Adachi et al. | May 2018 | A1 |
| 20180151796 | Akahane | May 2018 | A1 |
| 20180172995 | Lee et al. | Jun 2018 | A1 |
| 20180188115 | Hsu et al. | Jul 2018 | A1 |
| 20180189568 | Powderly et al. | Jul 2018 | A1 |
| 20180190017 | Mendez et al. | Jul 2018 | A1 |
| 20180191990 | Motoyama et al. | Jul 2018 | A1 |
| 20180196084 | Tustaniwskyj | Jul 2018 | A1 |
| 20180217395 | Lin et al. | Aug 2018 | A1 |
| 20180218545 | Garcia et al. | Aug 2018 | A1 |
| 20180250589 | Cossairt et al. | Sep 2018 | A1 |
| 20180260218 | Gopal | Sep 2018 | A1 |
| 20180284877 | Klein | Oct 2018 | A1 |
| 20180292654 | Wall et al. | Oct 2018 | A1 |
| 20180299678 | Singer et al. | Oct 2018 | A1 |
| 20180357472 | Dreessen | Dec 2018 | A1 |
| 20190005069 | Filgueiras de Araujo et al. | Jan 2019 | A1 |
| 20190011691 | Peyman | Jan 2019 | A1 |
| 20190056591 | Tervo et al. | Feb 2019 | A1 |
| 20190087015 | Lam | Mar 2019 | A1 |
| 20190101758 | Zhu et al. | Apr 2019 | A1 |
| 20190107723 | Lee et al. | Apr 2019 | A1 |
| 20190137788 | Suen | May 2019 | A1 |
| 20190155034 | Singer et al. | May 2019 | A1 |
| 20190155439 | Mukherjee et al. | May 2019 | A1 |
| 20190158926 | Kang et al. | May 2019 | A1 |
| 20190162950 | Lapstun | May 2019 | A1 |
| 20190167095 | Krueger | Jun 2019 | A1 |
| 20190172216 | Ninan et al. | Jun 2019 | A1 |
| 20190178654 | Hare | Jun 2019 | A1 |
| 20190179654 | Hare | Jun 2019 | A1 |
| 20190182415 | Sivan | Jun 2019 | A1 |
| 20190196690 | Chong et al. | Jun 2019 | A1 |
| 20190206116 | Xu et al. | Jul 2019 | A1 |
| 20190219815 | Price et al. | Jul 2019 | A1 |
| 20190243123 | Bohn | Aug 2019 | A1 |
| 20190287270 | Nakamura et al. | Sep 2019 | A1 |
| 20190318502 | He et al. | Oct 2019 | A1 |
| 20190318540 | Piemonte et al. | Oct 2019 | A1 |
| 20190321728 | Imai et al. | Oct 2019 | A1 |
| 20190347853 | Chen et al. | Nov 2019 | A1 |
| 20190380792 | Poltaretskyi et al. | Dec 2019 | A1 |
| 20190388182 | Kumar et al. | Dec 2019 | A1 |
| 20200066045 | Stahl et al. | Feb 2020 | A1 |
| 20200098188 | Bar-Zeev et al. | Mar 2020 | A1 |
| 20200100057 | Galon et al. | Mar 2020 | A1 |
| 20200110928 | Al Jazaery | Apr 2020 | A1 |
| 20200117267 | Gibson | Apr 2020 | A1 |
| 20200117270 | Gibson | Apr 2020 | A1 |
| 20200184217 | Faulkner | Jun 2020 | A1 |
| 20200184219 | Magura et al. | Jun 2020 | A1 |
| 20200184653 | Faulker | Jun 2020 | A1 |
| 20200202759 | Ukai et al. | Jun 2020 | A1 |
| 20200242848 | Ambler et al. | Jul 2020 | A1 |
| 20200309944 | Thoresen et al. | Oct 2020 | A1 |
| 20200356161 | Wagner | Nov 2020 | A1 |
| 20200368616 | Delamont | Nov 2020 | A1 |
| 20200391115 | Leeper et al. | Dec 2020 | A1 |
| 20200409528 | Lee | Dec 2020 | A1 |
| 20210008413 | Asikainen | Jan 2021 | A1 |
| 20210033871 | Jacoby et al. | Feb 2021 | A1 |
| 20210041951 | Gibson | Feb 2021 | A1 |
| 20210053820 | Gurin et al. | Feb 2021 | A1 |
| 20210093391 | Poltaretskyi et al. | Apr 2021 | A1 |
| 20210093410 | Gaborit et al. | Apr 2021 | A1 |
| 20210093414 | Moore et al. | Apr 2021 | A1 |
| 20210097886 | Kuester et al. | Apr 2021 | A1 |
| 20210124901 | Liu et al. | Apr 2021 | A1 |
| 20210132380 | Wieczorek | May 2021 | A1 |
| 20210141225 | Meynen et al. | May 2021 | A1 |
| 20210142582 | Jones et al. | May 2021 | A1 |
| 20210158023 | Fu et al. | May 2021 | A1 |
| 20210158627 | Cossairt et al. | May 2021 | A1 |
| 20210173480 | Osterhout et al. | Jun 2021 | A1 |
| 20220366598 | Azimi et al. | Nov 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 100416340 | Sep 2008 | CN |
| 101449270 | Jun 2009 | CN |
| 102448566 | May 2012 | CN |
| 103460255 | Dec 2013 | CN |
| 104040410 | Sep 2014 | CN |
| 104603675 | May 2015 | CN |
| 105938426 | Sep 2016 | CN |
| 106662754 | May 2017 | CN |
| 107004303 | Aug 2017 | CN |
| 107683497 | Feb 2018 | CN |
| 109223121 | Jan 2019 | CN |
| 105190427 | Nov 2019 | CN |
| 0504930 | Mar 1992 | EP |
| 0535402 | Apr 1993 | EP |
| 0632360 | Jan 1995 | EP |
| 1215522 | Jun 2002 | EP |
| 13451682 | Feb 2003 | EP |
| 1494110 | Jan 2005 | EP |
| 1938141 | Jul 2008 | EP |
| 1943556 | Jul 2008 | EP |
| 2290428 | Mar 2011 | EP |
| 2350774 | Aug 2011 | EP |
| 1237067 | Jan 2016 | EP |
| 3139245 | Mar 2017 | EP |
| 3164776 | May 2017 | EP |
| 3236211 | Oct 2017 | EP |
| 2723240 | Aug 2018 | EP |
| 2896986 | Feb 2021 | EP |
| 2499635 | Aug 2013 | GB |
| 2542853 | Apr 2017 | GB |
| 938DEL2004 | Jun 2006 | IN |
| H03-036974 | Apr 1991 | JP |
| 09-121370 | May 1997 | JP |
| H10-333094 | Dec 1998 | JP |
| 2002-015222 | Jan 2002 | JP |
| 2002-529806 | Sep 2002 | JP |
| 2003-029198 | Jan 2003 | JP |
| 2003-141574 | May 2003 | JP |
| 2003-228027 | Aug 2003 | JP |
| 2003-329873 | Nov 2003 | JP |
| 2004-348169 | Dec 2004 | JP |
| 2005-151224 | Jun 2005 | JP |
| 2005-303843 | Oct 2005 | JP |
| 2007-012530 | Jan 2007 | JP |
| 2007-86696 | Apr 2007 | JP |
| 2007-273733 | Oct 2007 | JP |
| 2008-257127 | Oct 2008 | JP |
| 2009-090689 | Apr 2009 | JP |
| 2009-244869 | Oct 2009 | JP |
| 2010-014443 | Jan 2010 | JP |
| 2010-139575 | Jun 2010 | JP |
| 2010-146030 | Jul 2010 | JP |
| 2010-271526 | Dec 2010 | JP |
| 2011-033993 | Feb 2011 | JP |
| 2011-257203 | Dec 2011 | JP |
| 2011-530131 | Dec 2011 | JP |
| 2012-015774 | Jan 2012 | JP |
| 2012-088777 | May 2012 | JP |
| 2012-235036 | Nov 2012 | JP |
| 2013-525872 | Jun 2013 | JP |
| 2013-206322 | Oct 2013 | JP |
| 2014-500522 | Jan 2014 | JP |
| 2014-90386 | May 2014 | JP |
| 2014-174366 | Sep 2014 | JP |
| 2014-192550 | Oct 2014 | JP |
| 2015-191032 | Nov 2015 | JP |
| 2016-502120 | Jan 2016 | JP |
| 2016-85463 | May 2016 | JP |
| 2016-516227 | Jun 2016 | JP |
| 2016-126134 | Jul 2016 | JP |
| 2017-015697 | Jan 2017 | JP |
| 2017-108444 | Jun 2017 | JP |
| 2017-153498 | Sep 2017 | JP |
| 2017-531840 | Oct 2017 | JP |
| 2017-535825 | Nov 2017 | JP |
| 6232763 | Nov 2017 | JP |
| 2018-503165 | Feb 2018 | JP |
| 6333965 | May 2018 | JP |
| 2018-173739 | Nov 2018 | JP |
| 2019-177134 | Oct 2019 | JP |
| 2005-0010775 | Jan 2005 | KR |
| 10-2006-0059992 | Jun 2006 | KR |
| 10-2011-0006408 | Jan 2011 | KR |
| 10-1372623 | Mar 2014 | KR |
| 10-2017-0017243 | Feb 2017 | KR |
| 201219829 | May 2012 | TW |
| 201803289 | Jan 2018 | TW |
| 1991000565 | Jan 1991 | WO |
| 2000030368 | Jun 2000 | WO |
| 2002071315 | Sep 2002 | WO |
| 2004095248 | Nov 2004 | WO |
| 2006132614 | Dec 2006 | WO |
| 2007037089 | Apr 2007 | WO |
| 2007041678 | Apr 2007 | WO |
| 2007085682 | Aug 2007 | WO |
| 2007102144 | Sep 2007 | WO |
| 2008148927 | Dec 2008 | WO |
| 2009101238 | Aug 2009 | WO |
| 2010015807 | Feb 2010 | WO |
| 2014203440 | Dec 2010 | WO |
| 2012030787 | Mar 2012 | WO |
| 2013049012 | Apr 2013 | WO |
| 2013062701 | May 2013 | WO |
| 2013145536 | Oct 2013 | WO |
| 2014033306 | Mar 2014 | WO |
| 2015079610 | Jun 2015 | WO |
| 2015143641 | Oct 2015 | WO |
| 2015194597 | Dec 2015 | WO |
| 2016054092 | Apr 2016 | WO |
| 2017004695 | Jan 2017 | WO |
| 2017044761 | Mar 2017 | WO |
| 2017049163 | Mar 2017 | WO |
| 2017051595 | Mar 2017 | WO |
| 2017120475 | Jul 2017 | WO |
| 2017176861 | Oct 2017 | WO |
| 2017203201 | Nov 2017 | WO |
| 2018008232 | Jan 2018 | WO |
| 2018031261 | Feb 2018 | WO |
| 2018022523 | Feb 2018 | WO |
| 2018044537 | Mar 2018 | WO |
| 2018039273 | Mar 2018 | WO |
| 2018057564 | Mar 2018 | WO |
| 2018085287 | May 2018 | WO |
| 2018087408 | May 2018 | WO |
| 2018097831 | May 2018 | WO |
| 2018166921 | Sep 2018 | WO |
| 2018236587 | Dec 2018 | WO |
| 2019040493 | Feb 2019 | WO |
| 2019148154 | Aug 2019 | WO |
| 2020010226 | Jan 2020 | WO |
| Entry |
|---|
| “Communication Pursuant to Rule 164(1) EPC mailed on Feb. 23, 2022”, European Patent Application No. 20753144.3, (11 pages). |
| “Extended European Search Report issued on Aug. 24, 2022”, European Patent Application No. 20846338.0, (13 pages). |
| “Extended European Search Report issued on Aug. 8, 2022”, European Patent Application No. 19898874.3, (8 pages). |
| “Extended European Search Report issued on Sep. 8, 2022”, European Patent Application No. 20798769.4, (13 pages). |
| “FS_XR5G: Permanent document, v0.4.0”, Qualcomm Incorporated, 3GPP TSG-SA 4 Meeting 103 retrieved from the Internet: URL:http://www.3gpp.org/ftp/Meetings%5F3GP P%5FSYNC/SA4/Docs/S4%2DI90526%2Ezip [retrieved on Apr. 12, 2019], Apr. 12, 2019, (98 pages). |
| “Non Final Office Action mailed on Sep. 19, 2022”, U.S. Appl. No. 17/263,001, (14 pages). |
| “Second Office Action mailed on Jul. 13, 2022 with English Translation”, Chinese Patent Application No. 201880079474.6, (10 pages). |
| “Second Office Action mailed on Jun. 20, 2022 with English Translation”, Chinese Patent Application No. 201880089255.6, (14 pages). |
| Anonymous , “Koi Pond: Top iPhone App Store Paid App”, https://web.archive.org/web/20080904061233/https://www.iphoneincanada.ca/reviews /koi-pond-top-iphone-app-store-paid-app/—[retrieved on Aug. 9, 2022], (2 pages). |
| “Extended European Search Report issued on Dec. 14, 2022”, European Patent Application No. 20886547.7, (8 pages). |
| “Extended European Search Report mailed on Nov. 3, 2022”, European Patent Application No. 20770244.0, (23 pages). |
| “Final Office Action mailed on Dec. 29, 2022”, U.S. Appl. No. 17/098,059, (32 pages). |
| “First Office Action mailed on Sep. 16, 2022 with English translation”, Chinese Patent Application No. 201980063642.7, (7 pages). |
| “Non Final Office Action mailed on Dec. 7, 2022”, U.S. Appl. No. 17/357,795, (63 pages). |
| “Notice of Reason for Rejection mailed on Oct. 28, 2022 with English translation”, Japanese Patent Application No. 2020-531452, (3 pages). |
| “Office Action mailed on Nov. 24, 2022 with English Translation”, Japanese Patent Application No. 2020-533730, (11 pages). |
| “Decision of Rejection mailed on Jan. 5, 2023 with English translation”, Chinese Patent Application No. 201880079474.6, (10 pages). |
| “Final Office Action mailed on Mar. 10, 2023”, U.S. Appl. No. 17/357,795, (15 pages). |
| “First Office Action mailed on Dec. 22, 2022 with English translation”, Chinese Patent Application No. 201980061450.2, (11 pages). |
| “First Office Action mailed on Jan. 24, 2023 with English translation”, Japanese Patent Application No. 2020-549034, (7 pages). |
| “First Office Action mailed on Jan. 30, 2023 with English translation”, Chinese Patent Application No. 201980082951.9, (5 pages). |
| “First Office Action mailed on Mar. 27, 2023 with English translation”, Japanese Patent Application No. 2020-566617, (6 pages). |
| “First Office Action mailed on Mar. 6, 2023 with English translation”, Korean Patent Application No. 10-2020-7019685, (7 pages). |
| “Non Final Office Action mailed on Apr. 13, 2023”, U.S. Appl. No. 17/098,043, (7 pages). |
| “Non Final Office Action mailed on Feb. 3, 2023”, U.S. Appl. No. 17/429,100, (16 pages). |
| “Non Final Office Action mailed on Feb. 3, 2023”, U.S. Appl. No. 17/497,965, (32 pages). |
| “Non Final Office Action mailed on Jan. 24, 2023”, U.S. Appl. No. 17/497,940, (10 pages). |
| “Non Final Office Action mailed on Mar. 1, 2023”, U.S. Appl. No. 18/046,739, (34 pages). |
| “Office Action mailed on Mar. 30, 2023 with English translation”, Japanese Patent Application No. 2020-566620, (10 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on May 30, 2022”, European Patent Application No. 19768418.6, (6 pages). |
| “Extended European Search Report issued on Jul. 20, 2022”, European Patent Application No. 19885958.9, (9 pages). |
| “Extended European Search Report mailed on May 30, 2022”, European Patent Application No. 20753144.3, (10 pages). |
| “Final Office Action mailed on Jul. 13, 2022”, U.S. Appl. No. 17/262,991, (18 pages). |
| “First Examination Report Mailed on Jul. 27, 2022”, Chinese Patent Application No. 201980036675.2, (5 pages). |
| “First Examination Report Mailed on Jul. 28, 2022”, Indian Patent Application No. 202047024232, (6 pages). |
| “Non Final Office Action mailed on Jul. 26, 2022”, U.S. Appl. No. 17/098,059, (28 pages). |
| Chittineni, C. , et al., “Single filters for combined image geometric manipulation and enhancement”, Proceedings of SPIE vol. 1903, Image and Video Processing, Apr. 8, 1993, San Jose, CA. (Year. 1993), pp. 111-121. |
| “Communication Pursuant to Article 94(3) EPC mailed on Feb. 28, 2023”, European Patent Application No. 19845418.3, (6 Pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Jul. 28, 2023”, European Patent Application No. 19843487.0, (15 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on May 23, 2023”, European Patent Application No. 18890390.0, (5 pages). |
| “Extended European Search Report issued on Apr. 5, 2023”, European Patent Application No. 20888716.6, (11 pages). |
| “Final Office Action mailed Oct. 16, 2023”, U.S. Appl. No. 17/098,043, (7 pages). |
| “Final Office Action mailed on Sep. 8, 2023 with English translation”, Japanese Patent Application No. 2020-566620, (18 pages). |
| “First Examination Report Mailed on Aug. 8, 2023”, Australian Patent Application No. 2018379105, (3 pages). |
| “First Office Action mailed Apr. 21, 2023 with English translation”, Japanese Patent Application No. 2021-509779, (26 pages). |
| “First Office Action mailed Jul. 4, 2023 with English translation”, Japanese Patent Application No. 2021-505669, (6 pages). |
| “First Office Action mailed on Apr. 13, 2023 with English Translation”, Japanese Patent Application No. 2020-567766, (7 pages). |
| “First Office Action mailed on Jun. 13, 2023 with English translation”, Japanese Patent Application No. 2020-567853, (7 pages). |
| “First Office Action mailed on May 26, 2023 with English translation”, Japanese Patent Application No. 2021-500607, (6 pages). |
| “First Office Action mailed on May 30, 2023 with English translation”, Japanese Patent Application No. 2021-519873, (8 pages). |
| “First Office Action mailed Sep. 29, 2023 with English translation”, Japanese Patent Application No. 2023-10887, (5 pages). |
| “Non Final Office Action mailed on Jul. 20, 2023”, U.S. Appl. No. 17/650,188, (11 pages). |
| “Non Final Office Action mailed on Jun. 14, 2023”, U.S. Appl. No. 17/516,483, (10 pages). |
| “Non Final Office Action mailed on May 11, 2023”, U.S. Appl. No. 17/822,279, (24 pages). |
| “Non Final Office Action mailed on Oct. 11, 2023”, U.S. Appl. No. 17/357,795, (14 pages). |
| “Non Final Office Action mailed on Oct. 24, 2023”, U.S. Appl. No. 17/259,020, (21 pages). |
| “Notice of Allowance mailed on Jul. 27, 2023 with English translation”, Korean Patent Application No. 10-2020-7019685, (4 pages). |
| “Office Action mailed on Apr. 13, 2023 with English translation”, Japanese Patent Application No. 2020-533730, (13 pages). |
| “Office Action mailed on Jul. 20, 2023 with English translation”, Japanese Patent Application No. 2021-505884, (6 pages). |
| “Office Action mailed on Jun. 8, 2023 with English translation”, Japanese Patent Application No. 2021-503762, (6 pages). |
| “Second Office Action mailed on May 2, 2023 with English Translation”, Japanese Patent Application No. 2020-549034, (6 pages). |
| “Second Office Action mailed on Sep. 25, 2023 with English translation”, Japanese Patent Application No. 2020-567853, (8 pages). |
| “Wikipedia Dioptre”, Jun. 22, 2018 (Jun. 22, 2018), XP093066995, Retrieved from the Internet: URL:https://en.wikipedia.org/w/index.php? title=Dioptre&direction=next&oldid=846451540 [retrieved on Jul. 25, 2023], (3 pages). |
| Li, Yujia , et al., “Graph Matching Networks for Learning the Similarity of Graph Structured Objects”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, XP081268608, Apr. 29, 2019. |
| Luo, Zixin , et al., “ContextDesc: Local Descriptor Augmentation With Cross-Modality Context”, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, XP033686823, DOI: 10.1109/CVPR.2019.00263 [retrieved on Jan. 8, 2020], Jun. 15, 2019, pp. 2522-2531. |
| Zhang, Zen , et al., “Deep Graphical Feature Learning for the Feature Matching Problem”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, XP033723985, DOI: 10.1109/ICCV.2019.00519 [retrieved on Feb. 24, 2020], Oct. 27, 2019, pp. 5086-5095. |
| “ARToolKit: Hardware” https://web.archive.org/web/20051013062315/http://www.hitl.washington.edu:80/artoolkit/documentation/hardware.htm (downloaded Oct. 26, 2020), Oct. 13, 2015, (3 pages). |
| “Communication according to Rule 164(1) EPC mailed on Feb. 23, 2022” , European Patent Application No. 20753144.3 , (11 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Sep. 4, 2019” , European Patent Application No. 10793707.0 , (4 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Apr. 25, 2022” , European Patent Application No. 18885707.2 , (5 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Jan. 4, 2022” , European Patent Application No. 20154070.5 , (8 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Oct. 21, 2021”, European Patent Application No. 16207441.3 , (4 pages). |
| “Communication Pursuant to Rule 164(1) EPC mailed on Jul. 27, 2021”, European Patent Application No. 19833664.6 , (11 pages). |
| “European Search Report mailed on Oct. 15, 2020”, European Patent Application No. 20180623.9 , (10 pages). |
| “Extended European Search Report issued on May 20, 2020”, European Patent Application No. 20154070.5 , (7 pages). |
| “Extended European Search Report issued on Jan. 22, 2021”, European Patent Application No. 18890390.0 , (11 pages). |
| “Extended European Search Report issued on Nov. 3, 2020”, European Patent Application No. 18885707.2 , (7 pages). |
| “Extended European Search Report issued on Jun. 30, 2021”, European Patent Application No. 19811971.1 , (9 pages). |
| “Extended European Search Report issued on Mar. 4, 2021”, European Patent Application No. 19768418.6 , (9 pages). |
| “Extended European Search Report issued on Nov. 4, 2020”, European Patent Application No. 20190980.1 , (14 pages). |
| “Extended European Search Report mailed on Jun. 12, 2017”, European Patent Application No. 16207441.3 , (8 pages). |
| “Extended European Search Report mailed on Jan. 28, 2022”, European Patent Application No. 19815876.8 , (9 pages). |
| “Extended European Search Report mailed on Jan. 4, 2022”, European Patent Application No. 19815085.6 , (9 pages). |
| “Extended European Search Report mailed on Jul. 16, 2021”, European Patent Application No. 19810142.0 , (14 pages). |
| “Extended European Search Report mailed on Jul. 30, 2021”, European Patent Application No. 19839970.1 , (7 pages). |
| “Extended European Search Report mailed on Jun. 19, 2020”, European Patent Application No. 20154750.2 , (10 pages). |
| “Extended European Search Report mailed on Mar. 22, 2022”, European Patent Application No. 19843487.0 , (14 pages). |
| “Extended European Search Report mailed on May 16, 2022”, European Patent Application No. 19871001.4 , (9 pages). |
| “Extended European Search Report mailed on Oct. 27, 2021”, European Patent Application No. 19833664.6 , (10 pages). |
| “Extended European Search Report mailed on Sep. 20, 2021”, European Patent Application No. 19851373.1 , (8 pages). |
| “Extended European Search Report mailed on Sep. 28, 2021”, European Patent Application No. 19845418.3 , (13 pages). |
| “Final Office Action mailed on Aug. 10, 2020”, U.S. Appl. No. 16/225,961 , (13 pages). |
| “Final Office Action mailed on Dec. 4, 2019”, U.S. Appl. No. 15/564,517 , (15 pages). |
| “Final Office Action mailed on Feb. 19, 2020”, U.S. Appl. No. 15/552,897 , (17 pages). |
| “Final Office Action mailed on Feb. 23, 2022”, U.S. Appl. No. 16/748,193 , (23 pages). |
| “Final Office Action mailed on Feb. 3, 2022”, United States U.S. Appl. No. 16/864,721 , (36 pages). |
| “Final Office Action mailed on Jun. 15, 2021”, United States U.S. Appl. No. 16/928,313 , (42 pages). |
| “Final Office Action mailed on Mar. 1, 2021”, U.S. Appl. No. 16/214,575 , (29 pages). |
| “Final Office Action mailed on Mar. 19, 2021”, U.S. Appl. No. 16/530,776 , (25 pages). |
| “Final Office Action mailed on Nov. 24, 2020”, U.S. Appl. No. 16/435,933 , (44 pages). |
| “Final Office Action mailed on Sep. 17, 2021”, U.S. Appl. No. 16/938,782 , (44 pages). |
| “First Examination Report Mailed on May 13, 2022”, Indian Patent Application No. 202047026359 , (8 pages). |
| “First Office Action mailed on Feb. 11, 2022 with English translation”, Chinese Patent Application No. 201880089255.6 , (17 pages). |
| “First Office Action mailed on Mar. 14, 2022 with English translation”, Chinese Patent Application No. 201880079474.6 , (11 pages). |
| “International Search Report and Written Opinion mailed on Feb. 12, 2021”, International Application No. PCT/US20/60555 , (25 pages). |
| “International Search Report and Written Opinion mailed on Mar. 12, 2020”, International PCT Patent Application No. PCT/US19/67919 , (14 pages). |
| “International Search Report and Written Opinion mailed on Aug. 15, 2019”, International PCT Patent Application No. PCT/US19/33987 , (20 pages). |
| “International Search Report and Written Opinion mailed on Jun. 15, 2020”, International PCT Patent Application No. PCT/US2020/017023 , (13 pages). |
| “International Search Report and Written Opinion mailed on Oct. 16, 2019”, International PCT Patent Application No. PCT/US19/43097 , (10 pages). |
| “International Search Report and Written Opinion mailed on Oct. 16, 2019”, International PCT Patent Application No. PCT/US19/36275 , (10 pages). |
| “International Search Report and Written Opinion mailed on Oct. 16, 2019”, International PCT Patent Application No. PCT/US19/43099 , (9 pages). |
| “International Search Report and Written Opinion mailed on Jun. 17, 2016”, International PCT Patent Application No. PCT/FI2016/050172 , (9 pages). |
| “International Search Report and Written Opinion mailed on Feb. 2, 2021”, International PCT Patent Application No. PCT/US20/60550 , (9 pages). |
| “International Search Report and Written Opinion mailed on Oct. 22, 2019”, International PCT Patent Application No. PCT/US19/43751 , (9 pages). |
| “International Search Report and Written Opinion mailed on Dec. 23, 2019”, International PCT Patent Application No. PCT/US19/44953 , (11 pages). |
| “International Search Report and Written Opinion mailed on May 23, 2019”, International PCT Patent Application No. PCT/US18/66514 , (17 pages). |
| “International Search Report and Written Opinion mailed on Sep. 26, 2019”, International PCT Patent Application No. PCT/US19/40544 , (12 pages). |
| “International Search Report and Written Opinion mailed on Aug. 27, 2019”, International PCT Application No. PCT/US2019/035245, (8 pages). |
| “International Search Report and Written Opinion mailed on Dec. 27, 2019”, International Application No. PCT/US19/47746 , (16 pages). |
| “International Search Report and Written Opinion mailed on Dec. 3, 2020”, International Patent Application No. PCT/US20/43596 , (25 pages). |
| “International Search Report and Written Opinion mailed on Sep. 30, 2019”, International Patent Application No. PCT/US19/40324 , (7 pages). |
| “International Search Report and Written Opinion mailed on Sep. 4, 2020”, International Patent Application No. PCT/US20/31036 , (13 pages). |
| “International Search Report and Written Opinion mailed on Jun. 5, 2020”, International Patent Application No. PCT/US20/19871 , (9 pages). |
| “International Search Report and Written Opinion mailed on Aug. 8, 2019”, International PCT Patent Application No. PCT/US2019/034763 , (8 pages). |
| “International Search Report and Written Opinion mailed on Oct. 8, 2019”, International PCT Patent Application No. PCT/US19/41151, (7 pages). |
| “International Search Report and Written Opinion mailed on Jan. 9, 2020”, International Application No. PCT/US19/55185 , (10 pages). |
| “International Search Report and Written Opinion mailed on Feb. 28, 2019”, International Patent Application No. PCT/US18/64686 , (8 pages). |
| “International Search Report and Written Opinion mailed on Feb. 7, 2020”, International PCT Patent Application No. PCT/US2019/061265 , (11 pages). |
| “International Search Report and Written Opinion mailed on Jun. 11, 2019”, International PCT Application No. PCT/US19/22620 , (7 pages). |
| “Invitation to Pay Additional Fees mailed Aug. 15, 2019”, International PCT Patent Application No. PCT/US19/36275 , (2 pages). |
| “Invitation to Pay Additional Fees mailed Sep. 24, 2020”, International Patent Application No. PCT/US2020/043596 , (3 pages). |
| “Invitation to Pay Additional Fees mailed on Oct. 22, 2019”, International PCT Patent Application No. PCT/US19/47746 , (2 pages). |
| “Invitation to Pay Additional Fees mailed on Apr. 3, 2020”, International Patent Application No. PCT/US20/17023 , (2 pages). |
| “Invitation to Pay Additional Fees mailed on Oct. 17, 2019”, International PCT Patent Application No. PCT/US19/44953 , (2 pages). |
| “multi-core processor”, TechTarget , 2013 , (1 page). |
| “Non Final Office Action mailed Nov. 19. 2019”, U.S. Appl. No. 16/355,611 , (31 pages). |
| “Non Final Office Action mailed on Apr. 1, 2022”, U.S. Appl. No. 17/256,961 , (65 pages). |
| “Non Final Office Action mailed on Apr. 11, 2022”, U.S. Appl. No. 16/938,782 , (52 pages). |
| “Non Final Office Action mailed on Apr. 12, 2022”, U.S. Appl. No. 17/262,991 , (60 pages). |
| “Non Final Office Action mailed on Aug. 21, 2019”, U.S. Appl. No. 15/564,517 , (14 pages). |
| “Non Final Office Action mailed on Aug. 4, 2021”, U.S. Appl. No. 16/864,721 , (21 pages). |
| “Non Final Office Action mailed on Feb. 2, 2022”, U.S. Appl. No. 16/783,866 , (8 pages). |
| “Non Final Office Action mailed on Jan. 26, 2021”, U.S. Appl. No. 16/928,313 , (33 pages). |
| “Non Final Office Action mailed on Jan. 27, 2021”, U.S. Appl. No. 16/225,961 (15 pages). |
| “Non Final Office Action mailed on Jul. 27, 2020”, U.S. Appl. No. 16/435,933 , (16 pages). |
| “Non Final Office Action mailed on Jul. 9, 2021”, U.S. Appl. No. 17/002,663 , (43 pages). |
| “Non Final Office Action mailed on Jul. 9, 2021”, U.S. Appl. No. 16/833,093 , (47 pages). |
| “Non Final Office Action mailed on Jun. 10, 2021”, U.S. Appl. No. 16/938,782 , (40 Pages). |
| “Non Final Office Action mailed on Jun. 17, 2020”, U.S. Appl. No. 16/682,911 , (22 pages). |
| “Non Final Office Action mailed on Jun. 19, 2020”, U.S. Appl. No. 16/225,961 (35 pages). |
| “Non Final Office Action mailed on Jun. 29, 2021”, U.S. Appl. No. 16/698,588 , (58 pages). |
| “Non Final Office Action mailed on Mar. 3, 2021”, U.S. Appl. No. 16/427,337 , (41 pages). |
| “Non Final Office Action mailed on Mar. 31, 2022”, U.S. Appl. No. 17/257,814 , (60 pages). |
| “Non Final Office Action mailed on Mar. 9, 2022”, U.S. Appl. No. 16/870,676 , (57 pages). |
| “Non Final Office Action mailed on May 10, 2022”, U.S. Appl. No. 17/140,921 , (25 pages). |
| “Non Final Office Action mailed on May 17, 2022”, U.S. Appl. No. 16/748,193 , (11 pages). |
| “Non Final Office Action mailed on May 26, 2021”, U.S. Appl. No. 16/214,575 , (19 pages). |
| “Non Final Office Action mailed on Nov. 19, 2019”, U.S. Appl. No. 16/355,611 , (31 pages). |
| “Non Final Office Action mailed on Nov. 5, 2020”, U.S. Appl. No. 16/530,776 , (45 pages). |
| “Non Final Office Action mailed on Oct. 22, 2019”, U.S. Appl. No. 15/859,277 , (15 pages). |
| “Non Final Office Action mailed on Sep. 1, 2020”, U.S. Appl. No. 16/214,575 , (40 pages). |
| “Non Final Office Action mailed on Sep. 20, 2021”, U.S. Appl. No. 17/105,848 , (56 pages). |
| “Non Final Office Action mailed on Sep. 29, 2021”, U.S. Appl. No. 16/748,193 , (62 pages). |
| “Notice of Allowance mailed on Mar. 25, 2020”, U.S. Appl. No. 15/564,517 (11 pages). |
| “Notice of Allowance mailed on Oct. 5, 2020”, U.S. Appl. No. 16/682,911 , (27 pages). |
| “Notice of Reason of Refusal mailed on Sep. 11, 2020 with English translation”, Japanese Patent Application No. 2019-140435 , (6 pages). |
| “Phototourism Challenge”, CVPR 2019 Image Matching Workshop. https://image matching-workshop. github.io. , (16 pages). |
| “Summons to attend oral proceedings pursuant to Rule 115(1) EPC mailed on Jul. 15, 2019”, European Patent Application No. 15162521.7, (7 pages). |
| Aarik, J. , et al. , “Effect of crystal structure on optical properties of TiO2 films grown by atomic layer deposition”, Thin Solid Films; Publication [online). May 19, 1998 [retrieved Feb. 19, 2020]. Retrieved from the Internet: <URL: https://www.sciencedirect.com/science/article/pii/S0040609097001351 ?via%3Dihub>; DOI: 10.1016/S0040-6090(97)00135-1; see entire document , (2 pages). |
| Altwaijry , et al. , “Learning to Detect and Match Keypoints with Deep Architectures” , Proceedings of the British Machine Vision Conference (BMVC), BMVA Press, Sep. 2016, [retrieved on Jan. 8, 2021 (Jan. 8, 2021)] < URL: http://www.bmva.org/bmvc/2016/papers/paper049/index.html >, en lire document, especially Abstract. |
| Arandjelović, Relja , et al. , “Three things everyone should know to improve object retrieval” , CVPR, 2012 , (8 pages). |
| AZOM , “Silica—Silicon Dioxide (SiO2)” , AZO Materials; Publication [Online]. Dec. 13, 2001 [retrieved Feb. 19, 2020]. Retrieved from the Internet: <URL: https://www.azom.com/article.aspx?Article1D=1114>. |
| Azuma, Ronald T. , “A Survey of Augmented Reality” , Presence: Teleoperators and Virtual Environments 6, 4 (Aug. 1997), 355-385; https://web.archive.org/web/20010604100006/http://www.cs.unc.edu/˜azuma/ARpresence.pdf (downloaded Oct. 26, 2020). |
| Azuma, Ronald T. , “Predictive Tracking for Augmented Reality”, Department of Computer Science, Chapel Hill NC; TR95-007, Feb. 1995 , 262 pages. |
| Battaglia, Peter W, et al. , “Relational inductive biases, deep learning, and graph networks”, arXiv:1806.01261, Oct. 17, 2018 , pp. 1-40. |
| Berg, Alexander C , et al. , “Shape matching and object recognition using low distortion correspondences”, In CVPR, 2005 , (8 pages). |
| Bian, Jiawang , et al. , “GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence.”, In CVPR (Conference on Computer Vision and Pattern Recognition), 2017 , (10 pages). |
| Bimber, Oliver , et al. , “Spatial Augmented Reality: Merging Real and Virtual Worlds”, https://web.media.mit.edu/˜raskar/book/BimberRaskarAugmentedRealityBook.pdf; published by A K Peters/CRC Press (Jul. 31, 2005); eBook (3rd Edition, 2007) , (393 pages). |
| Brachmann, Eric , et al. , “Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses”, In ICCV (International Conference on Computer Vision ), arXiv:1905.04132v2 [cs.CV] Jul. 31, 2019 , (17 pages). |
| Butail , et al. , “Putting the fish in the fish tank: Immersive VR for animal behavior experiments”, In: 2012 IEEE International Conference on Robotics and Automation. May 18, 2012 (May 18, 2012) Retrieved on Nov. 14, 2020 (Nov. 14, 2020) from <http:/lcdcl.umd.edu/papers/icra2012.pdf> entire document. |
| Caetano, Tiberio S , et al. , “Learning graph matching”, IEEE TPAMI, 31(6):1048-1058, 2009. |
| Cech, Jan , et al. , “Efficient sequential correspondence selection by cosegmentation”, IEEE TPAMI, 32(9):1568-1581, Sep. 2010. |
| Cuturi, Marco , “Sinkhorn distances: Lightspeed computation of optimal transport”, NIPS, 2013 , (9 pages). |
| Dai, Angela , et al. , “ScanNet: Richly-annotated 3d reconstructions of indoor scenes”, In CVPR, arXiv:1702.04405v2 [cs.CV] Apr. 11, 2017 , (22 pages). |
| Deng, Haowen , et al. , “PPFnet: Global context aware local features for robust 3d point matching” , In CVPR, arXiv:1802.02669v2 [cs.CV] Mar. 1, 2018 , (12 pages). |
| Detone, Daniel , et al. , “Deep image homography estimation”, In RSS Work-shop: Limits and Potentials of Deep Learning in Robotics, arXiv:1606.03798v1 [cs.CV] Jun. 13, 2016 , (6 pages). |
| Detone, Daniel , et al. , “Self-improving visual odometry”, arXiv:1812.03245, Dec. 8, 2018 , (9 pages). |
| Detone, Daniel , et al. , “SuperPoint: Self-supervised interest point detection and description”, In CVPR Workshop on Deep Learning for Visual SLAM, arXiv:1712.07629v4 [cs.CV] Apr. 19, 2018 , (13 pages). |
| Dusmanu, Mihai , et al. , “D2-net: A trainable CNN for joint detection and description of local features”, CVPR, arXiv:1905.03561v1 [cs.CV] May 9, 2019 , (16 pages). |
| Ebel, Patrick , et al. , “Beyond cartesian representations for local descriptors”, ICCV, arXiv:1908.05547v1 [cs.CV] Aug. 15, 2019 , (11 pages). |
| Fischler, Martin A , et al. , “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography” , Communications of the ACM, 24(6): 1981 , pp. 381-395. |
| Gilmer, Justin , et al. , “Neural message passing for quantum chemistry”, In ICML, arXiv:1704.01212v2 [cs.LG] Jun. 12, 2017 , (14 pages). |
| Giuseppe, Donato , et al. , “Stereoscopic helmet mounted system for real time 3D environment reconstruction and indoor ego—motion estimation” , Proc. SPIE 6955, Head- and Helmet-Mounted Displays XIII: Design and Applications, SPIE Defense and Security Symposium, 2008, Orlando, Florida, United States , 69550P. |
| Goodfellow , “Titanium Dioxide—Titania (TiO2)”, AZO Materials; Publication [online]. Jan. 11, 2002 [retrieved Feb. 19, 2020]. Retrieved from the Internet: <URL: https://www.azom.com/article.aspx?Article1D=1179>. |
| Hartley, Richard , et al. , “Multiple View Geometry in Computer Vision” , Cambridge University Press, 2003, pp. 1-673. |
| Jacob, Robert J.K. , “Eye Tracking in Advanced Interface Design” , Human-Computer Interaction Lab, Naval Research Laboratory, Washington, D.C., date unknown. 2003 , pp. 1-50. |
| Lee , et al. , “Self-Attention Graph Pooling”, Cornell University Library/Computer Science/Machine Learning, Apr. 17, 2019 [retrieved on Jan. 8, 2021 from the Internet<URL: https://arxiv.org/abs/1904.08082 >, entire document. |
| Lee, Juho , et al. , “Set transformer: A frame-work for attention-based permutation-invariant neural networks”, ICML, arXiv:1810.00825v3 [cs.LG] May 26, 2019 , (17 pages). |
| Leordeanu, Marius , et al. , “A spectral technique for correspondence problems using pairwise constraints”, Proceedings of (ICCV) International Conference on Computer Vision, vol. 2, pp. 1482-1489, Oct. 2005 , (8 pages). |
| Levola, T. , “Diffractive Optics for Virtual Reality Displays”, Journal of the SID Eurodisplay 14/05, 2005, XP008093627, chapters 2-3, Figures 2 and 10 , pp. 467-475. |
| Levola, Tapani , “Invited Paper: Novel Diffractive Optical Components for Near to Eye Displays—Nokia Research Center”, SID 2006 DIGEST, 2006 SID International Symposium, Society for Information Display, vol. XXXVII, May 24, 2005, chapters 1-3, figures 1 and 3 , pp. 64-67. |
| Li, Yujia , et al. , “Graph matching networks for learning the similarity of graph structured objects” , ICML, arXiv:1904.12787v2 [cs.LG] May 12, 2019 , (18 pages). |
| Li, Zhengqi , et al. , “Megadepth: Learning single- view depth prediction from internet photos”, In CVPR, fromarXiv: 1804.00607v4 [cs.CV] Nov. 28, 2018 , (10 pages). |
| Libovicky , et al. , “Input Combination Strategies for Multi-Source Transformer Decoder”, Proceedings of the Third Conference on Machine Translation (WMT). vol. 1: Research Papers, Belgium, Brussels, Oct. 31-Nov. 1, 2018; retrieved on Jan. 8, 2021 (Jan. 8, 2021 ) from < URL: https://doi.org/10.18653/v1/W18-64026 >, entire document. |
| Loiola, Eliane Maria, et al. , “A survey for the quadratic assignment problem”, European journal of operational research, 176(2): 2007 , pp. 657-690. |
| Lowe, David G , “Distinctive image features from scale- invariant keypoints”, International Journal of Computer Vision, 60(2): 91-110, 2004 , (28 pages). |
| Luo, Zixin , et al. , “ContextDesc: Local descriptor augmentation with cross-modality context”, CVPR, arXiv:1904.04084v1 [cs.CV] Apr. 8, 2019 , (14 pages). |
| Memon, F. , et al. , “Synthesis, Characterization and Optical Constants of Silicon Oxycarbide”, EPJ Web of Conferences; Publication [online). Mar. 23, 2017 [retrieved Feb. 19, 2020) .<URL: https://www.epj-conferences.org/articles/epjconf/pdf/2017/08/epjconf_nanop2017 _00002.pdf>; DOI: 10.1051/epjconf/201713900002 , (8 pages). |
| Molchanov, Pavlo , et al. , “Short-range FMCW monopulse radar for hand-gesture sensing” , 2015 IEEE Radar Conference (RadarCon) (2015) , pp. 1491-1496. |
| Mrad , et al. , “A framework for System Level Low Power Design Space Exploration”, 1991. |
| Munkres, James , “Algorithms for the assignment and transportation problems”, Journal of the Society for Industrial and Applied Mathematics, 5(1): 1957 , pp. 32-38. |
| Ono, Yuki , et al. , “LF-Net: Learning local features from images” , 32nd Conference on Neural Information Processing Systems (NIPS 2018), arXiv:1805.09662v2 [cs.CV] Nov. 22, 2018, (13 pages). |
| Paszke, Adam , et al. , “Automatic differentiation in Pytorch”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA , (4 pages). |
| Peyré, Gabriel , et al. , “Computational Optimal Transport”, Foundations and Trends in Machine Learning, 11(5-6):355-607, 2019; arXiv:1803.00567v4 [stat.ML] Mar. 18, 2020 , (209 pages). |
| Qi, Charles Ruizhongtai, et al. , “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA., Jun. 7, 2017 , (10 pages). |
| Qi, Charles R , et al. , “Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation”, CVPR, arXiv:1612.00593v2 [cs.CV] Apr. 10, 2017 , (19 pages). |
| Radenović, Filip , et al. , “Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking”, CVPR, arXiv:1803.11285v1 [cs.CV] Mar. 29, 2018 , (10 pages). |
| Raguram, Rahul , et al. , “A comparative analysis of ransac techniques leading to adaptive real-time random sample consensus”, Computer Vision—ECCV 2008, 10th European Conference on Computer Vision, Marseille, France, Oct. 12-18, 2008, Proceedings, Part I , (15 pages). |
| Ranftl, René, et al. , “Deep fundamental matrix estimation”, European Conference on Computer Vision (ECCV), 2018 , (17 pages). |
| Revaud, Jerome , et al. , “R2D2: Repeatable and Reliable Detector and Descriptor”, In NeurIPS, arXiv:1906.06195v2 [cs.CV] Jun. 17, 2019 , (12 pages). |
| Rocco, Ignacio , et al. , “Neighbourhood Consensus Networks”, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, arXiv:1810.10510v2 [cs.CV] Nov. 29, 2018 , (20 pages). |
| Rublee, Ethan , et al. , “ORB: An efficient alternative to SIFT or SURF” , Proceedings of the IEEE International Conference on Computer Vision. 2564-2571. 2011; 10.1109/ICCV.2011.612654 , (9 pages). |
| Sarlin , et al. , “SuperGlue: Learning Feature Matching with Graph Neural Networks”, Cornell University Library/Computer Science/ Computer Vision and Pattern Recognition, Nov. 26, 2019 [retrieved on Jan. 8, 2021 from the Internet< URL: https://arxiv.org/abs/1911.11763 >, entire document. |
| Sattler, Torsten , et al. , “SCRAMSAC: Improving RANSAC's efficiency with a spatial consistency filter”, ICCV, 2009: 2090-2097., (8 pages). |
| Schonberger, Johannes Lutz, et al. , “Pixelwise view selection for un- structured multi-view stereo”, Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oct. 11-14, 2016, Proceedings, Part III, pp. 501-518, 2016. |
| Schonberger, Johannes Lutz, et al. , “Structure-from-motion revisited” , Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104-4113 , (11 pages). |
| Sheng, Liu , et al. , “Time-multiplexed dual-focal plane head-mounted display with a liquid lens”, Optics Letters, Optical Society of Amer I Ca, US, vol. 34, No. 11, Jun. 1, 2009 (Jun. 1, 2009), XP001524475, ISSN: 0146-9592 , pp. 1642-1644. |
| Sinkhorn, Richard , et al. , “Concerning nonnegative matrices and doubly stochastic matrices.” , Pacific Journal of Mathematics, 1967, pp. 343-348. |
| Spencer, T. , et al. , “Decomposition of poly(propylene carbonate) with UV sensitive iodonium 11 salts”, Polymer Degradation and Stability; (online]. Dec. 24, 2010 (retrieved Feb. 19, 2020]. , (17 pages). |
| Tanriverdi, Vildan , et al. , “Interacting With Eye Movements in Virtual Environments”, Department of Electrical Engineering and Computer Science, Tufts University; Proceedings of the SIGCHI conference on Human Factors in Computing Systems, Apr. 2000 , pp. 1-8. |
| Thomee, Bart , et al. , “YFCC100m: The new data in multimedia research”, Communications of the ACM, 59(2):64-73, 2016; arXiv: 1503.01817v2 [cs.MM] Apr. 25, 2016 , (8 pages). |
| Torresani, Lorenzo , et al. , “Feature correspondence via graph matching: Models and global optimization”, Computer Vision—ECCV 2008, 10th European Conference on Computer Vision, Marseille, France, Oct. 12-18, 2008, Proceedings, Part II , (15 pages). |
| Tuytelaars, Tinne , et al. , “Wide baseline stereo matching based on local, affinely invariant regions”, BMVC, 2000 , pp. 1-14. |
| Ulyanov, Dmitry , et al. , “Instance normalization: The missing ingredient for fast stylization”, arXiv: 1607.08022v3 [cs.CV] Nov. 6, 2017 , (6 pages). |
| Vaswani, Ashish , et al. , “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; arXiv:1706.03762v5 [cs.CL] Dec. 6, 2017 , (15 pages). |
| Velićković, Petar , et al. , “Graph attention networks”, ICLR, arXiv:1710.10903v3 [stat.ML] Feb. 4, 2018 , (12 pages). |
| Mllani, Cédric , “Optimal transport: old and new”, vol. 338. Springer Science & Business Media, Jun. 2008 , pp. 1-998. |
| Wang, Xiaolong , et al. , “Non-local neural networks” , CVPR, arXiv:1711.07971v3 [cs.CV] Apr. 13, 2018 , (10 pages). |
| Wang, Yue , et al. , “Deep Closest Point: Learning representations for point cloud registration”, ICCV, arXiv:1905.03304v1 [cs.CV] May 8, 2019 , (10 pages). |
| Wang, Yue , et al. , “Dynamic Graph CNN for learning on point clouds”, ACM Transactions on Graphics, arXiv:1801.07829v2 [cs.CV] Jun. 11, 2019 , (13 pages). |
| Weissel , et al. , “Process cruise control: event-driven clock scaling for dynamic power management”, Proceedings of the 2002 international conference on Compilers, architecture, and synthesis for embedded systems. Oct. 11, 2002 (Oct. 11, 2002) Retrieved on May 16, 2020 (May 16, 2020) from <URL: https://dl.acm.org/doi/pdf/10.1145/581630.581668>. |
| Yi, Kwang Moo, et al. , “Learning to find good correspondences”, CVPR, arXiv:1711.05971v2 [cs.CV] May 21, 2018 , (13 pages). |
| Yi, Kwang Moo , et al. , “Lift: Learned invariant feature transform”, ECCV, arXiv:1603.09114v2 [cs.CV] Jul. 29, 2016 , (16 pages). |
| Zaheer, Manzil , et al. , “Deep Sets”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; arXiv: 1703.06114v3 [cs.LG] Apr. 14, 2018 , (29 pages). |
| Zhang, Jiahui , et al. , “Learning two-view correspondences and geometry using order-aware network”, ICCV; aarXiv:1908.04964v1 [cs.CV] Aug. 14, 2019 , (11 pages). |
| Zhang, Li , et al. , “Dual graph convolutional net- work for semantic segmentation”, BMVC, 2019; arXiv:1909.06121v3 [cs.CV] Aug. 26, 2020 , (18 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Oct. 6, 2023”, European Patent Application No. 19851373.1, (6 pages). |
| “Extended European Search Report issued on Jan. 8, 2024”, European Patent Application No. 23195266.4, (8 pages). |
| “Final Office Action mailed on Dec. 1, 2023”, U.S. Appl. No. 17/357,795, (18 pages). |
| “First Office Action mailed Dec. 12, 2023 with English translation”, Japanese Patent Application No. 2021-545712, (8 pages). |
| “First Office Action mailed Dec. 20, 2023 with English translation”, Chinese Patent Application No. 201980050600.X, (21 pages). |
| “First Office Action mailed Dec. 27, 2023 with English translation”, Chinese Patent Application No. 201980075942.7, (7 pages). |
| “First Office Action mailed Nov. 2, 2023 with English translation”, Chinese Patent Application No. 201980090867.1, (16 pages). |
| “First Office Action mailed on Dec. 11, 2023 with translation”, Chinese Patent Application No. 201980032005.3, (17 pages). |
| “First Office Action mailed on Dec. 25, 2023 with English translation”, Chinese Patent Application No. 2019800046303.8, (13 pages). |
| “Non Final Office Action mailed on Feb. 26, 2024”, U.S. Appl. No. 18/046,739, (48 pages). |
| “Non Final Office Action mailed on Nov. 22, 2023”, U.S. Appl. No. 17/268,376, (8 pages). |
| “Non Final Office Action mailed on Nov. 3, 2023”, U.S. Appl. No. 17/416,248, (17 pages). |
| “Office Action mailed Nov. 21, 2023 with English Translation”, Japanese Patent Application No. 2021-535716, (15 pages). |
| “Office Action mailed on Dec. 14, 2023 with English translation”, Japanese Patent Application No. 2021-526564, (13 pages). |
| “Office Action mailed on Nov. 7, 2023 with English translation”, Korean Patent Application No. 10-2023-7036734, (5 pages). |
| “Office Action mailed on Nov. 8, 2023 with English translation”, Chinese Patent Application No. 201980060018.1, (12 pages). |
| “Penultimate Office Action mailed on Oct. 19, 2023 with English translation”, Japanese Patent Application No. 2021-509779, (5 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Feb. 21, 2024”, European Patent Application No. 20770244.0, (8 pages). |
| “Extended European Search Report issued on Apr. 25, 2024”, European Patent Application No. 23208907.8, (9 pages). |
| “Final Office Action mailed on May 24, 2024”, U.S. Appl. No. 18/046,739, (52 pages). |
| “First Office Action mailed Mar. 1, 2024 with English translation”, Japanese Patent Application No. 2021-553297, (5 pages). |
| “First Office Action mailed on Feb. 1, 2024 with English translation”, Chinese Patent Application No. 202080018865.4, (9 pages). |
| “First Office Action mailed on Mar. 25, 2024 with English translation”, Chinese Patent Application No. 202080018919.7, (21 pages). |
| “Non Final Office Action mailed on May 16, 2024”, U.S. Appl. No. 18/361,546, (11 pages). |
| “Office Action mailed on Feb. 19, 2024 with English translation”, Korean Patent Application No. 10-2020-7020552, (18 pages). |
| “Office Action mailed on Feb. 26, 2024 with English translation”, Chinese Patent Application No. 201980069194.1, (11 pages). |
| “Office Action mailed on Mar. 6, 2024 with English translation”, Chinese Patent Application No. 201980053016.X, (7 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Mar. 11, 2024”, European Patent Application No. 20798769.4, (12 pages). |
| “Extended European Search Report issued on Aug. 6, 2024”, European Patent Application No. 24184599.9, (14 pages). |
| “Extended European Search Report issued on Jul. 9, 2024”, European Patent Application No. 24166847.4, (8 pages). |
| “First Office Action mailed Jun. 20, 2024 with English translation”, Japanese Patent Application No. 2021-564496, (14 pages). |
| “First Office Action mailed Jun. 24, 2024 with English translation”, Japanese Patent Application No. 2022-504602, (7 pages). |
| “First Office Action mailed Mar. 20, 2024 with English translation”, Chinese Patent Application No. 202080048293.4, (22 pages). |
| “Non Final Office Action mailed on Jun. 17, 2024”, U.S. Appl. No. 18/348,732, (19 pages). |
| “Non Final Office Action mailed on Sep. 24, 2024”, U.S. Appl. No. 18/597,716, (9 pages). |
| “Second Office Action with English translation mailed on Jul. 2, 2024”, Chinese Patent Application No. 201980032005.3, (15 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed Nov. 28, 2024”, European Patent Application No. 19885958.9, (5 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Nov. 21, 2024”, European Patent Application No. 20846338.0, (11 pages). |
| “Extended European Search Report issued on Dec. 2, 2024”, European Patent Application No. 24167829.1, (7 pages). |
| “First Office Action mailed Oct. 17, 2024 with English translation”, Japanese Patent Application No. 2022-527990, (24 pages). |
| “Office Action mailed on Nov. 7, 2024 with English translation”, Korean Patent Application No. 10-2024-7032937, (7 pages). |
| “Penultimate Office Action mailed on Sep. 17, 2024 with English translation”, Japanese Patent Application No. 2023-115047, (7 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Feb. 7, 2025”, European Patent Application No. 20154070.5, (7 pages). |
| “Communication Pursuant to Article 94(3) EPC mailed on Nov. 14, 2024”, European Patent Application No. 19845418.3, (4 pages). |
| “Final Office Action mailed on Dec. 13, 2024 with English translation”, Japanese Patent Application No. 2021-564496, (13 pages). |
| “Non Final Office Action mailed on Jan. 3, 2025”, U.S. Appl. No. 18/746,709, (31 pages). |
| “Office Action mailed on Jan. 22, 2025 with English translation”, Japanese Patent Application No. 2024-63271, (8 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20220337899 A1 | Oct 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62841806 | May 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16864721 | May 2020 | US |
| Child | 17807600 | US |
